Surgical needle key point positioning method, medical system, and medium

Abstract

The specification provides a surgical needle key point positioning method, a medical system and a medium. The surgical needle key point positioning method can include: training a to-be-processed picture containing a surgical instrument and a surgical needle, obtaining a network model, and outputting at least a heat map matching the number of surgical needle key points through the network model; performing smoothing processing on the heat map to obtain an adjusted heat map, and obtaining surgical needle key point coordinates on the adjusted heat map based on the adjusted heat map; mapping the surgical needle key point coordinates on the adjusted heat map to the to-be-processed picture to determine the surgical needle key point coordinates in the to-be-processed picture. The application can accurately position the target object in the image collected in a complex scene such as a surgical scene, especially a surgical needle with small pixels.

Description

Technical Field

[0001] This manual belongs to the field of medical device technology, and in particular relates to methods for locating key points of surgical needles, medical systems, and media. Background Technology

[0002] After acquiring endoscopic images of the surgical area, medical systems typically need to locate key points of target objects (e.g., surgical instruments, surgical needles) in the endoscopic images.

[0003] Currently, deep learning algorithms are mainly used in the field of image keypoint detection, employing convolutional neural networks to perform heatmap regression calculations on the coordinates of keypoints. However, this method is limited by computational complexity, resulting in unavoidable quantization errors in the keypoint detection results. This is particularly problematic when the image contains both relatively large objects like surgical instruments and smaller objects like surgical needles, severely impacting the accuracy of locating the smaller needle's keypoint.

[0004] Currently, there is an urgent need to improve existing methods for locating surgical needle key points in order to enhance the accuracy of locating surgical needle key points with relatively small pixels. Summary of the Invention

[0005] This manual provides a method, medical system, and medium for locating key points of a surgical needle, which can accurately locate key points of target objects, especially surgical needles with small pixels, in images acquired in complex environments such as surgical scenes.

[0006] This specification provides a method for locating key points of a surgical needle, including:

[0007] The network model is trained on the images to be processed, which contain surgical instruments and surgical needles, and the network model outputs a heat map that matches at least the number of key points of the surgical needles.

[0008] The heatmap is smoothed to obtain an adjusted heatmap, and the coordinates of the surgical needle key points on the adjusted heatmap are calculated based on the adjusted heatmap.

[0009] The coordinates of the surgical needle key points on the adjusted heatmap are mapped to the image to be processed, thereby determining the coordinates of the surgical needle key points in the image to be processed.

[0010] In a preferred embodiment, training the image to be processed, which includes surgical instruments and surgical needles, includes:

[0011] Data labels are generated by annotating key points of surgical needles in images containing surgical instruments and needles.

[0012] The data corresponding to the data labels are preprocessed to construct training data.

[0013] In a preferred embodiment, obtaining the network model includes:

[0014] A matching deep learning network model is selected based on the training data;

[0015] The training data is input into the deep learning network model for model training to obtain a network model for outputting heatmaps.

[0016] In a preferred embodiment, the network model outputs a scale map that can adjust for scale differences while also outputting a heatmap that matches the number of key points of the surgical needle.

[0017] In a preferred embodiment, the method further includes:

[0018] Add network branches to the network model;

[0019] Predict the target scale map using the network branches;

[0020] Multiply the standard deviation of the target scale map predicted by the network branch and the standard deviation of the heat map to obtain the standard deviation under different fields of view. Among them, the standard deviation corresponding to the surgical instrument key point with a larger effective area is greater than that of the surgical needle key point with a smaller effective area.

[0021] In a preferred embodiment, the method further includes:

[0022] The original Gaussian kernel size of the data labels is adjusted using the scaling map output by the network model, and the scaling coefficient of the original Gaussian kernel of the data labels is obtained to generate optimized data labels.

[0023] The loss function of the network model is determined based on the optimized data labels and the heatmap output by the network model.

[0024] The parameters of the network model are optimized inversely based on the loss function.

[0025] In a preferred embodiment, training the image to be processed, which includes surgical instruments and needles, specifically involves training it using a deep neural network.

[0026] In a preferred embodiment, the training steps of the deep neural network include:

[0027] Initialize the parameters and counters in the deep neural network;

[0028] The image to be processed is converted into an array and then input into the deep neural network.

[0029] The output of each layer of the deep neural network is obtained by calculation, and the error between the label and the output is determined by the loss function.

[0030] Based on the error determined by the loss function, the parameters of each layer of the deep neural network are updated by backpropagation using the chain derivative method.

[0031] Training ends when the error meets the preset conditions.

[0032] In a preferred embodiment, before training on the images to be processed containing surgical instruments and needles, the method further includes: acquiring images to be processed, the images to be processed including directly acquired images and / or optimized images.

[0033] In a preferred embodiment, when the image to be processed is an optimized image, the optimization process includes:

[0034] The acquired image is input into the image detection network model to obtain the detection box position of the target object in the image;

[0035] Based on the location of the detection box, the target object is extracted from the image to obtain an image containing the target object.

[0036] In a preferred embodiment, determining the coordinates of key points on the adjusted heatmap based on the adjusted heatmap includes:

[0037] The adjusted heatmap is represented using a two-dimensional Gaussian function;

[0038] Taking the logarithm of both sides of the two-dimensional Gaussian function above, we obtain the first relational expression;

[0039] Import the coordinates of key points on the heatmaps before and after adjustment into the Taylor formula to obtain the second relational expression;

[0040] The first and second derivatives are calculated based on the first and second relations to obtain the coordinates of key points on the adjusted heatmap.

[0041] A medical system includes an endoscope and an image processing device, the image processing device being used to process images acquired by the endoscope using the surgical needle key point localization method described above.

[0042] A computer-readable storage medium having computer instructions stored thereon, which, when executed, implement the relevant steps of any of the methods described above.

[0043] The features and advantages of this invention are:

[0044] Overall, the method for locating surgical needle key points provided in this application involves training an image containing surgical instruments and a surgical needle to obtain a matching network model. The heatmap output by the network model is then smoothed to more accurately calculate its peak value, thus optimizing the coordinates of the surgical needle key points. When the optimized coordinates are mapped onto the original image (i.e., the image to be processed), the coordinates of the surgical needle key points in the image can be determined more precisely. When applied to surgical instruments, it can more accurately locate the position of the surgical needle key points of small-pixel targets such as surgical needles. Therefore, the accuracy of the surgical needle key point coordinates is improved with lower computational costs, laying the foundation for subsequent automated surgical suturing.

[0045] Specific embodiments of the invention are disclosed in detail with reference to the following description and accompanying drawings, indicating how the principles of the invention can be employed. It should be understood that the embodiments of the invention are not therefore limited in scope. Within the spirit and scope of the appended claims, embodiments of the invention include many changes, modifications, and equivalents. Features described and / or shown for one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments. Attached Figure Description

[0046] To more clearly illustrate the embodiments described in this specification, the accompanying drawings used in the embodiments will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating the steps of a key point location method provided in one embodiment of this specification;

[0048] Figure 2 This is a schematic diagram illustrating one implementation of the key point localization method provided in this specification, applied in a scenario example.

[0049] Figure 3 This is a schematic diagram illustrating one implementation of the key point localization method provided in this specification, applied in a scenario example.

[0050] Figure 4 This is a flowchart illustrating a method for locating key points provided in one embodiment of this specification;

[0051] Figure 5 This is a schematic diagram illustrating one implementation of the key point localization method provided in this specification, applied in a scenario example.

[0052] Figure 6 This is a schematic diagram illustrating the algorithm principle of a key point localization method provided in the embodiments of this specification.

[0053] Figure 7 Is Figure 6 A diagram illustrating the data labels used in the algorithm;

[0054] Figure 8 This is a schematic diagram illustrating an optimization of heatmap labels provided in the embodiments of this specification;

[0055] Figure 9 This is a schematic diagram of the deep learning network in the key point localization method provided in the embodiments of this specification;

[0056] Figure 10 This is a schematic diagram of a network training process in one of the key point localization methods provided in the embodiments of this specification;

[0057] Figure 11 This is a schematic diagram of a loss function optimization method in the key point localization method provided in the embodiments of this specification;

[0058] Figure 12 This is a schematic diagram illustrating the mapping of a heat map to the original image in the key point location method provided in the embodiments of this specification;

[0059] Figure 13 This is a schematic diagram of the heatmap smoothing process in the key point location method provided in the embodiments of this specification.

[0060] Figure 14 This is a schematic diagram of the key point coordinate optimization in the key point positioning method provided in the embodiments of this specification;

[0061] Figure 15 A schematic diagram comparing the prediction results obtained using the key point localization method provided in the embodiments of this specification with the prediction results obtained using traditional methods;

[0062] Figure 16 This is a flowchart illustrating the application of a key point localization method provided in this specification during surgery.

[0063] Figure 17 This is a flowchart illustrating the application of another method for locating key points provided in this specification during surgery. Detailed Implementation

[0064] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0065] See Figure 1 As shown in the figure, this specification provides a method for locating key points of a surgical needle. Specifically, this method is applied to a medical system, and more specifically, to the field of surgical instruments. Specifically, it is a method for locating key points of a surgical instrument containing a small-pixel target—the surgical needle—thereby improving the accuracy of locating small-pixel targets within the surgical instrument. Of course, this method for locating key points of a surgical needle is not limited to the application areas exemplified above. This specification primarily uses its application in surgical instruments as an example for illustration. Other application scenarios can be adapted by referring to the examples provided in this specification, and will not be detailed further here.

[0066] In practice, this method may include the following:

[0067] Step S11: Train the image to be processed, which contains surgical instruments and surgical needles, to obtain a network model, and output a heat map that matches at least the number of key points of the surgical needles through the network model.

[0068] Step S13: Smooth the heat map to obtain an adjusted heat map, and calculate the coordinates of the surgical needle key points on the adjusted heat map based on the adjusted heat map.

[0069] Step S15: Map the coordinates of the surgical needle key points on the adjusted heatmap to the image to be processed, and determine the coordinates of the surgical needle key points in the image to be processed.

[0070] In some embodiments, the image to be processed, containing surgical instruments and needles, can be acquired using an image acquisition device, such as an endoscope. The image to be processed contains at least the target objects for which key point localization is required, such as surgical instruments and needles. The method of acquiring the image to be processed can include any of the following: directly acquired images, or images that have undergone optimization processing.

[0071] When the image to be processed is an image directly acquired through an image acquisition device such as an endoscope, the size of the image meets the set requirements (generally within a preset size range), which helps to reduce the workload of key point localization in subsequent images and improve the accuracy of key point localization. The specific value of this preset size range may vary depending on the specific application scenario, and this application does not impose a specific limitation on it.

[0072] When the image to be processed is an optimized image, the optimization methods include:

[0073] S101: Input the acquired image into the image detection network model to obtain the detection box position of the target object in the image;

[0074] S102: Based on the position of the detection box, the target object is extracted from the image to obtain an image containing the target object.

[0075] Taking the application of this method in a surgical platform as an example, images acquired using an endoscope may contain not only target objects, such as surgical instruments and needles, but also other non-target objects, such as other instruments on the surgical platform. These non-target objects serve as background to the target objects; the smaller their proportion in the image, the easier it is to reduce the difficulty of subsequent data processing and improve its accuracy. Therefore, a detection box can be defined for the target objects, and based on the position of this detection box, the target objects can be cropped within it to obtain images with a high proportion of the target objects, or even images containing only the target objects.

[0076] Of course, it should be noted that the shape of the detection box can be rectangular, circular, or other regular or even irregular shapes; this application does not impose any specific limitations on it. In some special cases, the detection box may also contain a small number of non-target object images.

[0077] In this embodiment, training the image to be processed, which includes surgical instruments and surgical needles, in step S11 may include:

[0078] S111: Generate data labels by annotating the key points of the surgical needle in the image to be processed, which contains surgical instruments and surgical needles.

[0079] S113: Preprocess the data corresponding to the data labels to construct training data.

[0080] Preprocessing may include any one or more combinations of the following: random cropping, horizontal flipping, and color dithering.

[0081] In some embodiments, the image processing method described above can be specifically applied to a medical system (e.g., a surgical robot). See [link to relevant documentation] for details. Figure 2 and Figure 3 .

[0082] in, Figure 2 Specifically, it refers to the control console 1 of the medical system, which is oriented towards the doctor and operated and controlled by the doctor. Figure 3 The operating table 2 of the medical system is a patient-facing device that responds to the operation commands initiated by the doctor through the console 1 to perform specific surgical procedures for the patient. The operating table 2 may be equipped with related instruments and equipment such as an endoscope 3, a robotic arm, an image processing device, and a robotic arm control device.

[0083] The aforementioned endoscope 3 may specifically include rigid endoscopes and flexible endoscopes. The endoscope 3 has components such as a camera and an image sensor. The camera is used to image the object being photographed onto the image sensor, and the image sensor is used to output an electrical signal based on the image from the camera. The electrical signal is used to generate an image containing the object being photographed (target).

[0084] For details, please refer to the following: Figure 4 This is a flowchart illustrating a method for locating key points of a surgical needle according to one embodiment of this application. Before performing this method for locating key points of the surgical needle, endoscopic images can be acquired. Figure 5 As shown, the Figure 5 The illustration shows a schematic diagram of an image under an endoscope. Under the endoscopic view, surgical instruments, surgical needles, and soft tissues in the abdominal cavity can be seen. The generated image is also the data sample used in the embodiments of this application.

[0085] After acquiring images from the endoscope, data labels can be created for the key points corresponding to the target objects in the images. Subsequently, the data can be preprocessed by random cropping, horizontal flipping, color jittering, etc., to construct training data.

[0086] The key point can be initially identified based on key locations of the target object, such as the tip of a surgical instrument, the head of a surgical needle, or the tail of a needle. Preprocessing the data in this data label can make the constructed training data richer, more realistic, and more comprehensive, which is more conducive to accurately reconstructing the target object in subsequent data processing.

[0087] Specifically, obtaining the network model in step S11 may include the following steps:

[0088] S115: Select a matching deep learning network model based on the training data;

[0089] S117: Input the training data into the deep learning network model for model training to obtain a network model for outputting heatmaps.

[0090] After completing data preprocessing and constructing training data, a deep learning network can be built and the network model trained. Specifically, such as... Figure 4 As shown, we can investigate deep learning network models that match the characteristics of this data, build a deep learning network, train the model using pre-prepared training data as input, and then use this network model to output a heatmap. The heatmap is the feature map extracted by the model at the end, also commonly known as a probability map. Figure 1 Generally, the size of the heatmap is in a predetermined ratio to the original input image size, for example, it can be 1 / 4 of the original image size. Of course, the size ratio of the heatmap to the original input image can be adjusted according to the actual application requirements, and this application does not make specific limitations here.

[0091] After obtaining the network model used to output the heatmap, steps S13 and S15 can be executed, namely, smoothing the heatmap to obtain an adjusted heatmap, and calculating the coordinates of key points on the adjusted heatmap based on the adjusted heatmap; mapping the coordinates of key points on the adjusted heatmap to the image to be processed to determine the coordinates of key points in the image to be processed.

[0092] The maximum value in the adjusted heatmap can be obtained using the heatmap output by the model. This maximum value corresponds to the coordinates of the key point in the heatmap. Since these coordinates are on the scale map, they can be mapped back to the original map to obtain the coordinates of the original map. Overall, the method for locating key points of surgical needles provided in this application improves the coordinate accuracy of key points of small targets such as surgical needles while maintaining low computational cost, laying the foundation for subsequent automated surgical suturing.

[0093] The inventors of this application also discovered that the traditional key point detection method uses a fixed standard deviation for heatmap regression, which ignores the scale differences of the target object under different viewing angles, thus affecting the accuracy of key point coordinate positioning for small targets such as surgical needles.

[0094] To address the aforementioned issues, the network model described in this application outputs a scale map that can adjust for scale differences while simultaneously outputting a heatmap that matches the number of key points, thereby improving the accuracy of coordinate positioning for small targets such as surgical needles.

[0095] Specifically, this key point localization method may also include:

[0096] S121: Add network branches to the network model;

[0097] S123: Predict the target scale map using the network branches;

[0098] S125: Multiply the standard deviation of the target scale map predicted by the network branch and the standard deviation of the heat map to obtain the standard deviation under different fields of view, wherein the standard deviation corresponding to the surgical instrument key point with a larger effective area is greater than that of the surgical needle key point with a smaller effective area.

[0099] See also Figure 4 The scale map is obtained by performing a convolution operation on the output of the backbone network. It is mainly used to optimize data labels and improve label accuracy.

[0100] In addition, based on the model output results, model testing can be performed. If the results are good, the model can be directly deployed in an engineering manner. If the results are not good, it can be further determined whether the problem is a data issue or a network issue, and further adaptive adjustments can be made subsequently.

[0101] In one specific implementation, with Figure 2 and Figure 3 Taking the scenario shown as an example, and referring to relevant materials... Figure 4 The method for locating key points of surgical instruments including the surgical needle (or simply the method for locating key points of the surgical needle) can be performed according to the following procedure:

[0102] A1. Acquire images under the endoscope, including color images of surgical instruments. The images can be saved in formats such as jpg and png.

[0103] A2. Through data annotation, generate corresponding original labels for heatmaps from the images acquired by the endoscope;

[0104] A3. Based on the combination of the original labels of the network output scale map and heat map, optimize the original labels and generate the final optimized data labels;

[0105] A4. Select an appropriate keypoint localization network based on the image characteristics, such as MSPN, OpenPose, AlphaPose, HRNetPose, etc.

[0106] A5. Input the prepared training data into the key point localization network to train the model;

[0107] A6. Optimize the model loss function;

[0108] A7. Use the model to output the coordinates of key points on the heatmap, and use coordinate decoding to restore the coordinates of key points on the heatmap to the coordinate space of the original image, thus obtaining the coordinates of key points in the original image (e.g., an image acquired by an endoscope).

[0109] The following section, in conjunction with specific accompanying drawings, elaborates and explains the method for locating key points provided in this application.

[0110] The method for locating this key point may also include:

[0111] S141: Adjust the original Gaussian kernel size of the data labels using the scaling graph output by the network model, obtain the scaling coefficient of the original Gaussian kernel of the data labels, and generate optimized data labels;

[0112] S143: Determine the loss function of the network model based on the optimized data labels and the heatmap output by the network model;

[0113] S145: Optimize the parameters of the network model in reverse according to the loss function.

[0114] Specifically, such as Figure 6 The diagram shown is a schematic diagram of the algorithm principle of the surgical instrument key point localization method according to an embodiment of the present invention, which can be divided into a training stage and an inference stage.

[0115] During the model training phase

[0116] 1. After the original image a1 is processed by the network model a2, a scale map and a heat map can be output.

[0117] Specifically, 2. The original Gaussian kernel size of the original label a3 is adjusted using the scaling graph. This size adjustment a4 mainly involves adjusting the scaling coefficient of the original Gaussian kernel of the original label a3, which is obtained adaptively through the network model a2. That is, the scaling graph is treated as a weighting coefficient, and the original label a3 is adjusted through the exponential algorithm to generate the latest optimized label a5.

[0118] 3. Calculate the loss function based on the latest optimized label a5 and the heatmap output by the model, and then optimize the model parameters in reverse based on the loss function.

[0119] During the model inference phase (a1->a2->heatmap->a6)

[0120] 1. The process is the same as step 1 of the model training stage (a1->a2->heatmap). Based on the output of the heatmap, the coordinates are decoded to restore the original coordinate space a6, thereby realizing the positioning of key points of surgical instruments.

[0121] like Figure 7 As shown, this is a schematic diagram of the data labeling method for the key point algorithm provided in this embodiment of the invention. The data labeling method is to obtain the A23 label map by taking the (x, y) coordinates of each key point (e.g., needle tip point, left and right support points, needle tail point, and pivot point) in A22 as the center and using a fixed value as the variance through a two-dimensional Gaussian function.

[0122] As shown in the A23 label image, the closer the distance to the center point (x, y), the higher the probability value, the larger the pixel value, and the whiter the corresponding image.

[0123] like Figure 8 As shown in the embodiments of this specification, a method for optimizing heatmap labels is provided. In the formula...

[0124] Representing the covariance matrix, x in the formula represents a two-dimensional random vector x = (xy), and S in the formula... * A two-dimensional diagonal matrix composed of scale graphs S

[0125] μ represents the center position of x, and σ0 represents the variance of x. Represents the original heatmap function. This represents the optimized heatmap function.

[0126] Generally, original heatmap labels use the same standard deviation. The key point detection algorithm provided in this embodiment of the invention is designed for different effective area sizes, that is, it detects both larger and smaller effective areas (e.g., instruments and surgical needles) simultaneously. Furthermore, the effective area of ​​instrument key points is much larger than that of surgical needles (needle tip, needle tail). Accordingly, the standard deviation σ0 of the larger effective area (e.g., instrument) can be set relatively larger, while the standard deviation σ0 of the effective area (e.g., needle) can be set relatively smaller.

[0127] In this invention, since the key points of the instrument and needle in the effective area are of different sizes, a single standard deviation is not used, which better solves the sample imbalance problem that occurs when detecting both large and small effective areas simultaneously. This invention uses a deep learning network to automatically learn and generate such a scale map S and optimize the standard deviation σ0. Compared to manually designing different standard deviations, this method is relatively simple and easy to generalize.

[0128] This invention uses a deep neural network to locate the key points of surgical instruments in endoscopic images. Figure 9 The diagram illustrates the deep learning network designed in this invention. The network maintains the high resolution of the feature map throughout the horizontal direction, so that small targets such as the needle tip and the needle tail will not lose information due to downsampling. In the vertical direction, multi-scale fusion is performed through repeated parallel convolutions to enhance the high-resolution representation features.

[0129] This invention utilizes a deep learning network to process multiple feature maps of different resolutions in parallel, allowing these maps to continuously interact and enhance semantic and location information. As shown in the figure, the dotted line represents the high-resolution feature output obtained by upsampling transposed convolution in the last layer of the network, as proposed in this invention. This outputs two tensors of different sizes: 2*128*128*5 and 2*64*64*5 (where 5 represents the number of keypoints). Since both large and small targets (needles) are used for keypoint localization in this experiment, the small target is output with a resolution of 128*128, which helps to pinpoint its precise location. The large target can be output with a resolution of 64*64. During the prediction phase, the two scales are averaged for the final prediction.

[0130] In one embodiment of the present invention, the training of the image to be processed, which includes surgical instruments and surgical needles, specifically involves training through a deep neural network.

[0131] Specifically, the training steps of the deep neural network may include:

[0132] S110: Initialize the parameters and counters in the deep neural network;

[0133] S112: Convert the image to be processed into an array and then input it into the deep neural network;

[0134] S114: The output of each layer of the deep neural network is calculated and obtained through the deep neural network, and the error between the label and the output is determined by the loss function;

[0135] S116: Based on the error determined by the loss function, the parameters of each layer of the deep neural network are updated by backpropagation using the chain rule method.

[0136] S118: Training ends when the error meets the preset conditions.

[0137] Please refer to the following: Figure 10 The specific steps involved in training this network may include the following:

[0138] A51. Initialize all parameters w (weight matrix) and b (bias vector) in the neural network, where the counter p is initialized to 1;

[0139] A52. Input samples, that is, after converting the collected endoscopic images into an array using OpenCV functions, input them into the neural network;

[0140] After A53 and A52, the output results are calculated by a neural network and compared with the true labels. The error between the two is calculated by a loss function.

[0141] A54. Based on the error obtained in A53, the reverse gradient is updated using the chain rule method, thereby updating the parameter values ​​w and b of each layer of the neural network.

[0142] A55. Through continuous updates via A54, the training ends when the minimum error is reached or the counter p reaches the maximum value of the epoch (the highest number of iterations).

[0143] like Figure 11 As shown in the figure, where P in the formula represents the probability value corresponding to the heatmap. σ represents the optimized label value, where the weight W adopts a loss function similar to cross-entropy, σ0 represents the variance of the heatmap, σ0·S represents the original Gaussian kernel after scale optimization, and S represents the scale map.

[0144] like Figure 12 As shown, mapping a heatmap to the original image can include the following steps:

[0145] A71. After Gaussian smoothing the heat map predicted by the model, the adjusted heat map as shown in A71 is generated.

[0146] A72, the m value is directly obtained from the heat map predicted by the model, while the present invention obtains the true μ value by fitting a Gaussian distribution. The μ(x,y) value is the coordinate of the key point on the heat map.

[0147] A73. The original maximum probability value in the heat map is m, which is a positive integer. However, the actual maximum value of μ may be a decimal, so a shift is required.

[0148] A74. After finding the true maximum probability value μ, zoom out μ proportionally (e.g., by 4 times) and map it back to the original image to get the coordinates of the key points in the original image.

[0149] like Figure 13 As shown, the first row is the Gaussian distribution map of the axis point, and the second row is the Gaussian distribution map of the clamping point. A711 is the heat map output by the neural network model. It can be seen from the figure that there are multiple peaks near the center point of the heat map. After smoothing A711, A712 is obtained.

[0150] The diagram marked A711 is a heatmap predicted by the model. This heatmap has multiple peaks near the center point and does not belong to a strictly Gaussian structure. A712 is generated by performing a Gaussian smoothing operation on the structure of A711, which conforms to a Gaussian structure and prepares for calculating the final μ later.

[0151] like Figure 14As shown, this invention provides a method for optimizing keypoint coordinates. Traditionally, keypoint coordinates are directly calculated from the maximum value in the predicted heatmap, which is the m value in this image. However, this invention primarily uses a Gaussian structure for inference, optimizing the m value to find the true keypoint coordinates μ. Specifically, this keypoint coordinate optimization method may include the following steps:

[0152] The process of determining the coordinates of key points on the adjusted heatmap based on the adjusted heatmap includes:

[0153] The adjusted heatmap is represented using a two-dimensional Gaussian function;

[0154] Taking the logarithm of both sides of the two-dimensional Gaussian function above, we obtain the first relational expression;

[0155] Import the coordinates of key points on the heatmaps before and after adjustment into the Taylor formula to obtain the second relational expression;

[0156] The first and second derivatives are calculated based on the first and second relations to obtain the coordinates of key points on the adjusted heatmap.

[0157] like Figure 14 As shown, the heatmap predicted by A721 is represented by a two-dimensional Gaussian function. Specifically, the adjusted heatmap is represented by a two-dimensional Gaussian function as follows:

[0158]

[0159] Where x represents the position of each pixel, μ represents the center position of the Gaussian structure, which is the coordinate of the key point; ∑ represents the covariance matrix.

[0160] A722 represents the logarithmic function of taking both sides of formula A721;

[0161] This is the Taylor formula expression, where x0 represents a point closer to x.

[0162]

[0163] A724

[0164] The result is obtained by substituting μ and m into A723 according to Taylor's formula, where m is the coordinate of the key point obtained directly from the predicted heat map.

[0165] Zhongtong A725 calculates the first and second derivatives based on A724 and A722:

[0166]

[0167]

[0168] P′(x)=P″(x)·(x-μ);

[0169] μ=m-(P′′(m)) -1 P′(m).

[0170] like Figure 15 As shown in the figure, this diagram compares the prediction results obtained by the key point localization method provided in this specification with the prediction results obtained by the traditional method. In A81, the dotted circles represent the instrument's key points, the needle tip and the needle tail. Figure 15 As can be seen, the needle head and tail coordinates obtained using the key point localization method provided in this application are closer to the label values ​​in A83, while the needle head and tail will be offset by a few pixels in A82 using the existing traditional method.

[0171] like Figure 16 The diagram shown is a flowchart illustrating the application of a method for locating key points of a surgical needle provided in this embodiment of the specification during surgery.

[0172] The intraoperative tissue and organ images acquired by the endoscope are used to locate key points of the surgical instruments including the surgical needle according to the present invention. In particular, the key point location of the surgical needle with relatively small pixels is achieved. Based on the location of the key points of the surgical needle, the motion control end of the surgical instrument can automatically operate the surgical instrument to perform automatic surgery.

[0173] like Figure 17 The diagram shown is a flowchart illustrating the application of another method for locating key points of the surgical needle provided in this specification during surgery.

[0174] The intraoperative tissue and organ images acquired by the endoscope are used to obtain the position information of the surgical instrument, needle tip and needle tail through the key point positioning method of the surgical instrument including the surgical needle of the present invention. According to the relative distance between the surgical instrument and the needle, the surgical instrument is used to clamp the area near the needle tail. At the same time, according to the position of the wound obtained by image segmentation, a suture point is defined with a spacing of 0.5cm, and automatic suturing operation is performed.

[0175] This specification also provides a medical system, which may include an endoscope and an image processing device, wherein the image processing device is used to process images acquired by the endoscope using the key point localization method provided in this specification.

[0176] The images acquired by the endoscope can specifically be endoscopic images taken during a surgical procedure.

[0177] The image processing device uses the surgical needle key point localization method provided in the embodiments of this specification to process the images acquired by the endoscope, and can obtain corresponding processing results. Specifically, the processing results can be the location of key points of surgical instruments identified in the images acquired by the endoscope.

[0178] This specification also provides a computing device, including a processor and a memory for storing processor-executable instructions. In specific implementations, the processor can execute relevant steps of a method for locating key points of a surgical needle according to the instructions.

[0179] This specification also provides a computer storage medium based on the above-described image processing method. The computer storage medium stores computer program instructions that, when executed, perform the following: training a network model on an image to be processed containing surgical instruments and a surgical needle; outputting a heatmap matching at least the number of key points on the surgical needle using the network model; smoothing the heatmap to obtain an adjusted heatmap; and determining the coordinates of the key points on the adjusted heatmap; mapping the coordinates of the key points on the adjusted heatmap to the image to be processed, thereby determining the coordinates of the key points on the surgical needle in the image to be processed.

[0180] In this embodiment, the storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), or memory card. The memory can be used to store computer program instructions. The network communication unit can be an interface configured according to standards specified in the communication protocol for network connection communication.

[0181] In this embodiment, the specific functions and effects implemented by the program instructions stored in the computer storage medium can be explained by comparison with other embodiments, and will not be repeated here.

[0182] It should be noted that the units, devices, or modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. For ease of description, the above devices are described by dividing them into various modules according to their functions. Of course, in implementing this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical, or other forms.

[0183] While this specification provides the steps of operation described in the embodiments or flowcharts, more or fewer steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or client product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0184] Those skilled in the art will also know that, besides implementing the controller using purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the controller function as logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers (PLCs), and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the devices within it used to implement various functions can also be considered structures within that hardware component. Alternatively, the devices used to implement various functions can be considered as both software modules implementing the method and structures within a hardware component.

[0185] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, classes, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0186] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of this specification can essentially be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, mobile terminal, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of this specification.

[0187] The various embodiments described in this specification are presented in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. This specification can be applied to numerous general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable electronic devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices, etc.

[0188] Although this specification has been described by way of embodiments, those skilled in the art will recognize that many variations and modifications are possible without departing from the spirit of this specification, and it is intended that the appended claims cover such variations and modifications without departing from the spirit of this specification.

Claims

1. A method for locating key points of a surgical needle, characterized in that, include: The images to be processed, which contain surgical instruments and surgical needles, are processed to obtain training data. The training data is used to train a model to obtain a network model. The network model outputs a heat map that matches at least the number of key points of the surgical needle. The network model outputs a heatmap that matches the number of surgical needle key points, and also outputs a scale map that can adjust the scale difference. The standard deviation of the scale map and the heatmap are multiplied to obtain the standard deviation under different fields of view. The standard deviation corresponding to the surgical instrument key points with larger effective areas is greater than that of the surgical needle key points with smaller effective areas. The heatmap is smoothed to obtain an adjusted heatmap, and the coordinates of the surgical needle key points on the adjusted heatmap are calculated based on the adjusted heatmap. The coordinates of the surgical needle key points on the adjusted heatmap are mapped to the image to be processed, thereby determining the coordinates of the surgical needle key points in the image to be processed.

2. The method for locating key points of a surgical needle according to claim 1, characterized in that, The training data obtained by processing the images containing surgical instruments and needles includes: Data labels are generated by annotating key points of surgical needles in images containing surgical instruments and needles. The data corresponding to the data labels are preprocessed to construct training data.

3. The method for locating key points of a surgical needle according to claim 1, characterized in that, The acquisition of the network model includes: Select a matching deep learning network model based on the training data; The training data is input into the deep learning network model for model training to obtain a network model for outputting heatmaps.

4. The method for locating key points of a surgical needle according to claim 1, characterized in that, The method further includes: Add network branches to the network model; Predict the target scale map using the network branches; The standard deviations of the target scale map predicted by the network branch and the heatmap are multiplied to obtain the standard deviations under different views.

5. The method for locating key points of a surgical needle according to claim 1, characterized in that, The method further includes: The original Gaussian kernel size of the data labels is adjusted using the scaling map output by the network model, and the scaling coefficient of the original Gaussian kernel of the data labels is obtained to generate optimized data labels. The loss function of the network model is determined based on the optimized data labels and the heatmap output by the network model. The parameters of the network model are optimized inversely based on the loss function.

6. The method for locating key points of a surgical needle according to claim 1, characterized in that, The specific method for training the model on the training data is to train it using a deep neural network.

7. The method for locating key points of a surgical needle according to claim 6, characterized in that, The training steps of the deep neural network include: Initialize the parameters and counters in the deep neural network; The image to be processed is converted into an array and then input into the deep neural network. The output of each layer of the deep neural network is obtained by calculation, and the error between the label and the output is determined by the loss function. Based on the error determined by the loss function, the parameters of each layer of the deep neural network are updated by backpropagation using the chain derivative method. Training ends when the error meets the preset conditions.

8. The method for locating key points of a surgical needle according to claim 1, characterized in that, Before processing the images containing surgical instruments and needles to obtain training data, the method further includes: acquiring images to be processed, which include directly acquired images and / or optimized images.

9. The method for locating key points of a surgical needle according to claim 8, characterized in that, When the image to be processed is an optimized image, the optimization processing methods include: The acquired image is input into the image detection network model to obtain the detection box position of the target object in the image; Based on the location of the detection box, the target object is extracted from the image to obtain an image containing the target object.

10. The method for locating key points of a surgical needle according to claim 1, characterized in that, The process of determining the coordinates of key points on the adjusted heatmap based on the adjusted heatmap includes: The adjusted heatmap is represented using a two-dimensional Gaussian function; Taking the logarithm of both sides of the two-dimensional Gaussian function above, we obtain the first relational expression; Import the coordinates of key points on the heatmaps before and after adjustment into the Taylor formula to obtain the second relational expression; The first and second derivatives are calculated based on the first and second relations to obtain the coordinates of key points on the adjusted heatmap.

11. A medical system comprising an endoscope and an image processing device, characterized in that, The image processing device is used to process the images acquired by the endoscope using the method for locating the key points of the surgical needle as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, It stores computer instructions that, when executed, implement the relevant steps of the method according to any one of claims 1 to 10.

Images