Coordinate positioning method and system for kidney stones based on preoperative CT and intraoperative ultrasound images

Through the improved nn-Unet model and adaptive optimization filtering algorithm, combined with NLM mean operation and morphological operation, the combination of preoperative CT and intraoperative ultrasound images is achieved, solving the problem of real-time positioning of kidney stones and improving the accuracy of the surgery Sex and safety.

CN118762005BActive Publication Date: 2024-11-12TONGJI HOSPITAL ATTACHED TO TONGJI MEDICAL COLLEGE HUAZHONG SCI TECH +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411240487.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-12
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

The prior art is difficult to combine preoperative CT images with intraoperative ultrasound images to achieve real-time positioning of kidney stones, resulting in insufficient accuracy and real-time feedback on the position of kidney stones during the operation.

Method used

The improved nn-Unet model and adaptive optimization filtering algorithm are used, combined with NLM mean operation and morphological operation, and the data of preoperative CT images and intraoperative ultrasound videos are obtained, real-time coordinates of kidney stones are calculated, and the fully automated real-time kidney stones are realized. position.

Benefits of technology

It improves the accuracy of kidney stone positioning and the accuracy of surgery, reduces dependence on manual operations, and enhances the safety and reliability of surgery.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762005B_ABST
    Figure CN118762005B_ABST
Patent Text Reader

Abstract

The present invention provides a method and system for positioning the coordinates of kidney stones based on preoperative CT and intraoperative ultrasound images, the method comprising: obtaining preoperative CT images and preprocessing them; inputting the preprocessed preoperative CT images into an improved nn-Unet model to obtain kidney mask information, and calculating stone reference size data based on the kidney mask information; obtaining intraoperative ultrasound video and performing adaptive optimization filtering algorithm preprocessing; combining NLM mean operation and morphological operation to obtain a complete echo group in the intraoperative ultrasound video; based on the stone reference size data, screening out kidney stones from the complete echo group, and obtaining the real-time intraoperative coordinates of kidney stones. The present invention combines preoperative CT and intraoperative ultrasound images, and utilizes deep learning neural network technology to realize the positioning of kidney stones and real-time intraoperative position output, thereby reducing the positioning error caused by patient movement during the operation, being able to complete stone information feedback in real time, and providing reliable technical support for clinical surgery.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more specifically, to a method and system for positioning kidney stone coordinates based on preoperative CT and intraoperative ultrasound images. Background Art

[0002] Kidney stones are a common urinary system disease that seriously affects the quality of life of patients. In order to improve the accuracy and effectiveness of treatment, modern medicine increasingly relies on the combination of preoperative imaging and intraoperative imaging technology.

[0003] Preoperative CT (computed tomography) imaging technology can provide high-resolution three-dimensional structural information of the kidneys and surrounding tissues, which helps doctors to accurately evaluate and plan the location, size and morphology of kidney stones before surgery. However, CT images are static and only reflect the situation at the moment of imaging. They cannot reflect the possible displacement of kidney stones during surgery in real time. Intraoperative ultrasound imaging technology has the advantage of real-time imaging and can provide dynamic images of the kidneys and stones during surgery, but its clarity needs to be improved. How to combine preoperative CT images with intraoperative ultrasound images to achieve real-time positioning of kidney stones during surgery and provide real-time feedback of stone information, thereby providing more reliable technical support for clinical surgery, is a direction worthy of research. Summary of the invention

[0004] In view of the technical problems existing in the prior art, the present invention provides a computer-assisted kidney stone coordinate positioning method and system based on preoperative CT and intraoperative ultrasound images, which can realize the real-time position information output of stones during clinical surgery and help promote the automation research of extracorporeal lithotripsy.

[0005] According to a first aspect of the present invention, a method for coordinate positioning of kidney stones based on preoperative CT and intraoperative ultrasound images is provided, comprising:

[0006] Obtain preoperative CT images and perform preprocessing;

[0007] Inputting the preprocessed preoperative CT image into the improved nn-Unet model to obtain kidney mask information, and calculating the stone reference size data based on the kidney mask information;

[0008] Acquire intraoperative ultrasound video and perform preprocessing with adaptive optimization filtering algorithm;

[0009] Combine NLM mean operation and morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video;

[0010] Based on the stone reference size data, kidney stones are screened out from the complete echo group, and the real-time intraoperative coordinates of the kidney stones are obtained.

[0011] According to a second aspect of the present invention, there is provided a kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images, comprising:

[0012] Data acquisition module, used to obtain preoperative CT images and intraoperative ultrasound videos;

[0013] A data preprocessing module, used for analyzing the preoperative CT image and performing preprocessing, and for analyzing the intraoperative ultrasound video and performing preprocessing using an adaptive optimization filtering algorithm;

[0014] A reference information acquisition module, used to extract kidney mask information from the pre-processed preoperative CT image using an improved nn-Unet model, and calculate stone reference size data based on the kidney mask information;

[0015] An intraoperative stone extraction module is used to combine NLM mean operation and morphological operation to obtain a complete echo group in the intraoperative ultrasound video; based on the stone reference size data, kidney stones are screened out from the complete echo group to obtain the real-time intraoperative coordinates of the kidney stones;

[0016] The output module is used to update the current position information of the kidney stone in real time and output the real-time intraoperative coordinates of the kidney stone.

[0017] According to a third aspect of the present invention, there is provided an electronic device comprising a memory and a processor, wherein the processor is used to implement the steps of the above-mentioned method for coordinate positioning of kidney stones based on preoperative CT and intraoperative ultrasound images when executing a computer management program stored in the memory.

[0018] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer management program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned method for coordinate positioning of kidney stones based on preoperative CT and intraoperative ultrasound images.

[0019] The present invention provides a computer-aided kidney stone coordinate positioning method, system, electronic device and storage medium based on preoperative CT and intraoperative ultrasound images, which utilizes a deep learning model and an optimized filtering algorithm to improve the accuracy of image processing and kidney stone positioning; by combining preoperative CT images and intraoperative ultrasound images, fully automated real-time positioning of kidney stones is achieved, reducing reliance on manual operations during surgery, helping doctors to perform precise operations during surgery, and improving the accuracy and safety of surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flow chart of a method for coordinate positioning of kidney stones based on preoperative CT and intraoperative ultrasound images provided in a certain embodiment of the present invention;

[0021] Figure 2A flow chart of a method for coordinate positioning of kidney stones based on preoperative CT and intraoperative ultrasound images provided by another embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the classic nn-Unet model structure;

[0023] Figure 4 A schematic diagram of the structure of an improved nn-Unet model provided in an embodiment of the present invention;

[0024] Figure 5 A block diagram of the structure of a kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images provided by a certain embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0026] Figure 7 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0027] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0028] Figure 1 A flow chart of a kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images provided by the present invention, such as Figure 1 As shown, the method includes steps S1 to S5.

[0029] S1, obtain the patient's preoperative CT image and perform preprocessing.

[0030] Specifically, the patient is scanned by a CT scanning device before surgery to obtain a preoperative CT image, and the preoperative CT image data is stored in DICOM format. Since the unit of CT scanning measurement is HU (Husfield unit), this unit measures the radioactivity density, that is, the data format of the DICOM file is HU value. Therefore, the DICOM file is read using the pydicom parsing library, and after parsing multiple frames of the DICOM file, a three-dimensional image matrix based on HU value expression can be extracted.

[0031] The HU values ​​of the three-dimensional image matrix obtained by parsing the DICOM file are standardized and converted into pixel values, and the corresponding pixel values ​​in the CT image are normalized and converted into a preset threshold range (for example, within the range of 0-2000), so as to obtain a three-dimensional image matrix corresponding to the preoperative CT image expressed based on pixel values, so as to facilitate further image processing and analysis.

[0032] S2, inputting the preprocessed preoperative CT image into the improved nn-Unet model to obtain kidney mask information, and calculating the stone reference size data based on the kidney mask information.

[0033] It is understandable that the nn-Unet network model is a classic deep learning model with the following structure: Figure 3 The nn-Unet model is derived from the U-Net model and is similar in structure to the U-Net model, including a typical encoder-decoder structure.

[0034] The nn-Unet model is an improved model based on the 3D U-Net cascade, designed specifically for medical image segmentation tasks, with a flexible architecture parameter adjustment mechanism and automatic hyperparameter optimization. In order to obtain a better kidney mask region extraction effect, before extracting kidney mask information, this embodiment needs to adjust and optimize the parameters of the nn-Unet model in advance and train it. Specifically, the parameter adjustment optimization includes:

[0035] 1. Feature encoder adjustment: increase the number of encoder network layers, for example, from the original 5 layers to 7 layers, to extract more features; adjust the number of decoder layers, for example, from the original 5 layers to 7 layers;

[0036] 2. Adjust the number of convolution kernels of each layer of the encoder network, increasing layer by layer, and the increasing number of convolution kernels of each layer of the encoder network is [32, 64, 128, 256, 512, 1024, 2048]; adjust the number of convolution kernels of each layer of the decoder network, decreasing by half, and the decreasing number of convolution kernels of each layer of the decoder network is [2048, 1024, 512, 256, 128,64, 32];

[0037] S203. The convolution kernel size of each network convolution layer is adjusted from the conventional 3x3 to 7x7 to improve the receptive field of the deep learning model.

[0038] S204, multi-scale feature fusion, a multi-scale feature aggregation module is introduced in each layer of the encoder network, combining features of different scales to improve the recognition ability of the deep learning model for stones of different sizes. Specifically, the multi-scale feature aggregation module performs feature aggregation at different scales by using the spatial pyramid pooling (SPP) layer. Specifically, the spatial pyramid pooling layer uses different pooling window sizes (including 1x1, 2x2 and 4x4) for pooling the preoperative CT image, and then connects and fuses the pooled features. The feature aggregation at different scales through the SPP layer is to enhance the recognition ability of the deep learning model for targets of different scales.

[0039] Figure 4 FIG. 1 is a schematic diagram of the improved nn-Unet model structure in this embodiment. Figure 4 As shown, the improved nn-Unet model includes a first-stage model (corresponding to Figure 4 stage1) and the second stage model (corresponding to Figure 4 In stage2, the first stage model includes an encoder and a decoder, and the second stage model includes a 3D U-net model; wherein:

[0040] The encoder includes n layers of networks connected in sequence, n≥7, each layer of the network includes a first convolution module and a feature aggregation module, the convolution kernel size of the first convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network increases exponentially; the first convolution module is used to perform a convolution operation on the feature map input from the previous layer of the network to extract local feature information of the image; the feature aggregation module is used to downsample the feature map output by the first convolution module, perform a pooling operation, and obtain an aggregated feature map;

[0041] The decoder includes n layers of networks connected in sequence and an output module, n≥7, each layer of the network includes a fully connected module, an upsampling module and a second convolution module, the convolution kernel size of the second convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network decreases by half; the fully connected module is used to perform a residual connection between the feature map of the previous layer of the network and the output of the corresponding layer of the encoder network; the upsampling module is used to perform a reverse convolution operation on the feature map output by the previous layer of the network, and upsample the image to obtain a higher resolution prediction result; the second convolution module is used to extract a reorganized high-order feature map from the output result of the upsampling module; the output module is a convolution layer with a convolution kernel size of 1x1, which is used to output the high-order feature map;

[0042] The second stage model is used to upsample the high-order feature map output by the first stage model to the size of the original image, and then input it into the 3D U-net model for segmentation at full resolution.

[0043] It is understandable that the output module of the last layer of the decoder is a 1x1 convolutional layer, which can reduce the number of output channels. The decoder and the encoder are connected by multiple information streams, so the underlying detailed information can flow directly to the decoder. By supplementing the detailed information, the network model has a better understanding of the feature map and can accurately identify and locate the target area.

[0044] Based on the improved nn-Unet model, patient CT images were used for training to extract kidney mask information from CT images. Specifically, 200 patient CT images and corresponding kidney mask information were used in the training process, and the original CT images were randomly rotated, scaled, translated, and flipped to increase data diversity. The training process used the Adam optimizer, and the training learning rate was set to 0.001. During training, 20% of the sample data was divided for verification and 80% of the sample data was used for training.

[0045] After the improved nn-Unet model is trained, the preoperative CT image to be detected is input into the improved nn-Unet model, and the corresponding kidney mask information can be output. The stone reference size data is calculated based on the kidney mask, including S201~S203:

[0046] S201, a kidney mask region can be obtained through kidney mask information, and within the kidney mask region, a stone region is detected by a threshold segmentation method, and a connected domain analysis is performed on each segmented stone region.

[0047] S202, calculating the volume V of each stone area based on the voxels, and calculating the stone diameter d according to the volume V;

[0048] S203, counting the volume V of each stone area and its corresponding stone diameter d as stone reference size data.

[0049] For example, the stone threshold is set to 354, and the stone threshold is a grayscale value. The stone threshold is used to separate the stone area where the kidney stone is located from other tissues, and the three-dimensional volume information of the kidney stone can be extracted through multiple preoperative CT images at different angles. Specifically, the HU value of the kidney mask area is converted, and the pixel point data is standardized to the range of (0, 2000); the cv2.connectedComponents function of opencv is called to perform connected domain analysis on the stone area identified by threshold segmentation to ensure that each connected area represents a complete stone.

[0050] More specifically, the stone volume data can be used to calculate the diameter information of the stone. Assuming that the stone is approximately spherical, the calculation method is as follows: Calculate the volume V of each stone area, where the volume unit is voxels. Based on the following formula, calculate the stone diameter d according to the volume V of the stone area:

[0051] ,

[0052] After the stone diameter d is obtained, the volume V of the stone area corresponding to each stone and its corresponding stone diameter d are recorded respectively and stored as stone reference size data.

[0053] S3, obtaining intraoperative ultrasound video and performing preprocessing with an adaptive optimization filtering algorithm.

[0054] It can be understood that after obtaining the preoperative CT image to obtain the stone reference size data: the volume V of the stone area and its corresponding stone diameter d, the intraoperative ultrasound video of the patient is obtained through the intraoperative B-ultrasound equipment and the corresponding AVI file is parsed. The ultrasound video data includes multiple continuous frames, each frame is a two-dimensional grayscale image, representing an ultrasound image at a specific time point. The ultrasound video data is saved as an AVI file, and the video parsing uses the opencv library to parse the AVI file saved as the ultrasound video and extract the video data frame by frame. The parsing process includes opening the AVI file, reading the frame rate raw data of the video frame, and cutting the ultrasound video into several single-frame images arranged in sequence with time nodes for subsequent image recognition processing.

[0055] Step S3 specifically includes sub-steps S301 to S303.

[0056] S301, obtain the intraoperative ultrasound video and parse its corresponding AVI file, read the frame rate raw data of the video frame, cut the intraoperative ultrasound video into single-frame ultrasound images according to the frame rate raw data, and perform Z-score normalization and grayscale conversion on each frame of the ultrasound image.

[0057] It is understandable that this step pre-processes the ultrasound image obtained by analyzing the ultrasound video, and uses an adaptive optimization filtering algorithm to smooth and preliminarily reduce noise. The pre-processing includes at least Z-score standardization and grayscale conversion of each frame of the extracted ultrasound image for subsequent image processing and feature point detection. Z-score standardization refers to mean-variance standardization of each frame of the image, so that the pixel value of each frame of the image has the characteristics of a mean of 0 and a standard deviation of 1. The mean-variance standardization is calculated as follows:

[0058] ,

[0059] in, is the pixel value of the pixel position (i, j) in the original ultrasound image, is the mean value of ultrasound pixel data, is the standard deviation of the ultrasound data. After Z-score standardization, each pixel value in the ultrasound image is transformed to a distribution with mean 0 and standard deviation 1.

[0060] S302, using an adaptive optimized median filtering algorithm to perform preliminary smoothing and noise reduction processing on the ultrasound image to remove salt and pepper noise and enhance the characteristics of the echo cluster in the ultrasound image.

[0061] It is understandable that after the Z-score standardization and grayscale conversion in step S301 are completed, this step uses an adaptive optimized median filter algorithm to perform preliminary smoothing and noise reduction on the ultrasound image. In order to better extract the strong echo group of kidney stones in the ultrasound image in the future, a custom optimization parameter is introduced on the basis of the ordinary median filter algorithm to enhance the extraction effect of the strong echo group of stones. The formula of the adaptive median filter optimization algorithm is:

[0062] ,

[0063] in, is the pixel value at position (x,y) after adaptive median filtering. It is the core operation of median filtering. The pixel values ​​within the range are calculated and their median is calculated. Centered on the (x,y) pixel The window size k is preferably set to 3. Indicates the position (x,y) Filter window size The pixel value set includes all pixel data in the filtering window and is used to calculate the median. It is a custom optimization coefficient used to enhance the filtering effect and strengthen the edge of the echo cluster area. The value is preferably set to 1.2.

[0064] S303: Use a Gaussian blur algorithm model to further smooth and reduce noise on the ultrasound image to reduce high-frequency noise of the ultrasound image.

[0065] It can be understood that this step uses the Gaussian blur algorithm model to further smooth the ultrasound image and reduce high-frequency noise. The Gaussian blur optimization formula is:

[0066] ,

[0067] in, is an adaptive weight parameter of ultrasound image, which is used to enhance the characteristics of strong echo clusters. The preferred setting is 1.5. (x, y) is the coordinates of the ultrasound image pixel. It refers to the pixel value at the pixel position (x, y) of the image after Gaussian blur optimization. is the standard deviation of Gaussian blur, which controls the degree of processing. Here, it is preferably set to 0.5 according to the characteristics of the ultrasound image itself.

[0068] S4, combining NLM mean operation and morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video.

[0069] It can be understood that after the preprocessing of step S3 is completed, the preprocessed ultrasound image is processed using the NLM mean processing algorithm, combined with morphological operations, to obtain a complete echo cluster in the ultrasound data, specifically including steps S401~S402.

[0070] S401, after reducing the high frequency noise, first use the non-local mean (NLM) processing algorithm to retain the ultrasound image details. The calculation method of the NLM processing algorithm is:

[0071] ,

[0072] in, is the weight parameter of the strong echo group in the ultrasound image, which is used to improve the recognition ability of the stone area. Set to 1.3. is the image processed by NLM at position The pixel value at . It is a point The search window is centered on The size of the matrix, S is set to 31. is the center pixel The Euclidean distance between pixel (i,j) and the search window. It is a filtering parameter used to control the degree of smoothing. Here it is adjusted to 15 based on the experimental results. It is a normalization factor to ensure that the pixel value of the output image is in the correct range. It needs to be dynamically generated according to the weight calculation of the strong echo group in the ultrasound image. The calculation method is:

[0073] .

[0074] S402, using a dilation operation and an erosion operation in morphological operations to process the ultrasound image, remove noise between echo clusters and remove disconnected echo clusters, and retain complete echo clusters in the ultrasound image.

[0075] It is understandable that after the NLM algorithm is processed, morphological operations are used to remove noise between echo clusters and echo clusters with broken connections, so that complete echo clusters are screened out. In this embodiment, the morphological operations refer to expansion and corrosion operations. The echo cluster is the presentation form of stones in the ultrasound image. The stones in the ultrasound image have high density and artifacts, and are usually presented in the form of strong echo clusters.

[0076] Specifically, the morphological dilation operation is used to expand the boundaries of the target area in the image, fill small holes and connect broken echo clusters. The ultrasound image processed by the NLM algorithm is expanded, and the expansion operation structure element is a 5x5 square matrix.

[0077] The morphological erosion operation is used to shrink the boundaries of the target area in the image and remove small noise points and artifacts. The image is eroded to prevent the loss of strong echo cluster data. The erosion operation structural element is a 3x3 square matrix.

[0078] Through the above morphological operation, the noise between echo clusters and the disconnected echo clusters can be effectively removed, making the area in the ultrasound image more coherent and clear, thereby retaining the complete echo clusters.

[0079] S5, based on the stone reference size data, screen out the kidney stones from the complete echo group and obtain the real-time intraoperative coordinates of the kidney stones.

[0080] In this step, kidney stones are screened from ultrasound data by comparing the stone information calculated in the preoperative CT image with the data of the strong echo group in the ultrasound image, and screening out the strong echo group that meets the reference size data range. The strong echo group screened out by size is the kidney stone. Specifically, steps S501 to S505 are included.

[0081] S501, using a threshold segmentation method, filter out strong echo clusters from each complete echo cluster in the ultrasound image according to a preset strong echo cluster grayscale threshold. For example, the strong echo cluster grayscale threshold is set to 265, and those above this threshold are marked as candidate strong echo clusters.

[0082] S502, performing connected domain analysis on each strongly echogenic cluster screened out from the ultrasound image, and calculating the area of ​​each strongly echogenic cluster.

[0083] In this step, the features of the strong echo cluster are extracted, and the connected domain analysis is performed on each candidate strong echo cluster in the ultrasound image to calculate the area of ​​each candidate strong echo cluster. The connected domain analysis is implemented using the connected component algorithm function cv2.connectedComponents() provided by opencv.

[0084] S503, the method for calculating the diameter of the strong echo group is that, since the shape of the stone is approximately spherical, it is assumed that the strong echo group in the ultrasound image is approximately circular, and the strong echo group is equivalent to a circle, and the diameter of the strong echo group is calculated according to the area of ​​the strong echo group, that is, the diameter of the minimum circumscribed circle of the strong echo group is used as the diameter of the strong echo group.

[0085] S504, using the stone diameter d obtained from the preoperative CT image as a constraint condition, and screening kidney stones from the strong echo clusters in the intraoperative ultrasound image. Specifically, the diameter of each of the strong echo clusters is matched with the stone diameter d in the stone reference size data. If the deviation between the diameter of the strong echo cluster and the stone diameter d is within a preset diameter tolerance range (e.g., ±10%), it is determined that the diameter is successfully matched, and the strong echo cluster with a successful diameter match is determined to be a kidney stone.

[0086] S505, taking the center coordinates of the echogenic mass whose diameter is successfully matched in the ultrasound image as the real-time intraoperative coordinates of the kidney stone.

[0087] In one possible implementation, in order to further increase the accuracy of real-time positioning of kidney stones, the volume V of the stone area obtained in the preoperative CT image can be further used as another constraint condition, and the kidney stones are double-screened and positioned by the diameter constraint condition and the volume constraint condition, as follows:

[0088] S506, obtaining the coordinates of the contour points of the strong echo mass in the ultrasound image, combining the coordinates of the contour points of the strong echo mass in multiple frames of ultrasound images at multiple angles, and constructing a three-dimensional model of the strong echo mass;

[0089] Since kidney stones are approximately spherical, it is assumed that the three-dimensional shape of the strong echo mass is spherical. The contour points of the strong echo mass in each frame of ultrasound images at different angles are located on the spherical surface of the strong echo mass. The three-dimensional model of the strong echo mass can be calculated based on the coordinates of multiple contour points on the spherical surface.

[0090] S507, calculating the volume of the strong echo cluster according to the three-dimensional model of the strong echo cluster;

[0091] Since the three-dimensional shape of the strong echo group is assumed to be spherical, the volume of the strong echo group can be calculated by calculating the volume of the sphere, and the coordinates of the center point of the three-dimensional model of the strong echo group can be obtained at the same time;

[0092] S508, matching the volume of each of the strong echo clusters with the stone volume V in the stone reference size data; if the deviation between the volume of the strong echo cluster and the stone volume V is within the volume tolerance range, then the volume matching is determined to be successful; otherwise, returning to step S503, recalculating the diameter of the strong echo cluster;

[0093] S509, compare the center coordinates of the three-dimensional model of the strong echo group with successful volume matching with the center coordinates of the strong echo group in the ultrasound image with successful diameter matching. If the difference between the two is within the coordinate tolerance range, output the center coordinates of the strong echo group in the ultrasound image as the real-time intraoperative coordinates of the kidney stone. Otherwise, return to step S503 and recalculate the diameter of the strong echo group.

[0094] Figure 5 A structural diagram of a kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images provided by an embodiment of the present invention, such as Figure 5 As shown, a kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images includes a data acquisition module, a data preprocessing module, a reference information acquisition module, an intraoperative stone extraction module and an output module, wherein:

[0095] Data acquisition module, used to obtain preoperative CT images and intraoperative ultrasound videos;

[0096] A data preprocessing module, used for analyzing the preoperative CT image and performing preprocessing, and for analyzing the intraoperative ultrasound video and performing preprocessing using an adaptive optimization filtering algorithm;

[0097] A reference information acquisition module, used to extract kidney mask information from the pre-processed preoperative CT image using an improved nn-Unet model, and calculate stone reference size data based on the kidney mask information;

[0098] An intraoperative stone extraction module is used to combine NLM mean operation and morphological operation to obtain a complete echo group in the intraoperative ultrasound video; based on the stone reference size data, kidney stones are screened out from the complete echo group to obtain the real-time intraoperative coordinates of the kidney stones;

[0099] The output module is used to update the current position information of the kidney stone in real time and output the real-time intraoperative coordinates of the kidney stone.

[0100] It can be understood that the kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images provided by the present invention corresponds to the kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images provided in the aforementioned embodiments. The relevant technical features of the kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images can refer to the relevant technical features of the kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images, which will not be repeated here.

[0101] See also Figure 6 , Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention provides an electronic device 600, including a memory 610, a processor 620, and a computer program 611 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 611, the following steps are implemented:

[0102] Obtain preoperative CT images and perform preprocessing;

[0103] Inputting the preprocessed preoperative CT image into the improved nn-Unet model to obtain kidney mask information, and calculating the stone reference size data based on the kidney mask information;

[0104] Acquire intraoperative ultrasound video and perform preprocessing with adaptive optimization filtering algorithm;

[0105] Combine NLM mean operation and morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video;

[0106] Based on the stone reference size data, kidney stones are screened out from the complete echo group, and the real-time intraoperative coordinates of the kidney stones are obtained.

[0107] See also Figure 7 , Figure 7 Schematic diagram of an embodiment of a computer-readable storage medium provided by the present invention. Figure 7 As shown, this embodiment provides a computer-readable storage medium 700, on which a computer program 711 is stored. When the computer program 711 is executed by a processor, the following steps are implemented:

[0108] Obtain preoperative CT images and perform preprocessing;

[0109] Inputting the preprocessed preoperative CT image into the improved nn-Unet model to obtain kidney mask information, and calculating the stone reference size data based on the kidney mask information;

[0110] Acquire intraoperative ultrasound video and perform preprocessing with adaptive optimization filtering algorithm;

[0111] Combine NLM mean operation and morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video;

[0112] Based on the stone reference size data, kidney stones are screened out from the complete echo group, and the real-time intraoperative coordinates of the kidney stones are obtained.

[0113] The embodiment of the present invention provides a kidney stone coordinate positioning method, system, electronic device and storage medium based on preoperative CT and intraoperative ultrasound images, which, by combining an improved nn-Unet neural network model, an adaptive optimized median filter algorithm, an NLM mean algorithm and a variety of morphological operations, uses preoperative CT images and intraoperative ultrasound videos to jointly locate kidney stones, and can obtain accurate position information of kidney stones in real time during the operation. By combining preoperative CT and intraoperative ultrasound images, using deep learning neural network technology, the positioning of kidney stones and real-time position output during the operation are achieved, the positioning error caused by patient movement during the operation is reduced, and the stone information feedback can be completed in real time, which provides reliable technical support for clinical surgery and helps to improve the treatment effect and patient experience.

[0114] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0116] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0117] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0118] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0119] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0120] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images, It is characterized in that include: Obtain preoperative CT images and perform preprocessing; The pre-processed preoperative CT image is input into the improved nn-Unet model to obtain kidney mask information, and the stone reference size data is calculated based on the kidney mask information; the improved nn-Unet model includes a first-stage model and a second-stage model which are arranged in sequence, the first-stage model includes an encoder and a decoder, and the second-stage model includes a 3DU-net model; wherein: The encoder includes n layers of networks connected in sequence, n≥7, each layer of the network includes a first convolution module and a feature aggregation module, the convolution kernel size of the first convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network increases exponentially; the first convolution module is used to perform a convolution operation on the feature map input from the previous layer of the network to extract local feature information of the image; the feature aggregation module is used to downsample the feature map output by the first convolution module, perform a pooling operation, and obtain an aggregated feature map; The decoder includes n layers of networks connected in sequence and an output module, n≥7, each layer of the network includes a fully connected module, an upsampling module and a second convolution module, the convolution kernel size of the second convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network decreases by half; the fully connected module is used to perform a residual connection between the feature map of the previous layer of the network and the output of the corresponding layer of the encoder; the upsampling module is used to perform a reverse convolution operation on the feature map output by the previous layer of the network, and upsample the image to obtain a prediction result with a higher resolution; the second convolution module is used to extract a reorganized high-order feature map from the output result of the upsampling module; the output module is used to output the high-order feature map; The second stage model is used to upsample the high-order feature map output by the first stage model to the size of the original image, and then input it into the 3D U-net model for segmentation at full resolution; Acquire intraoperative ultrasound video and perform preprocessing with adaptive optimization filtering algorithm; Combine NLM mean operation and morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video; Based on the stone reference size data, kidney stones are screened out from the complete echo group, and the real-time intraoperative coordinates of the kidney stones are obtained.

2. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 1, It is characterized in that The method of obtaining a preoperative CT image and performing preprocessing includes: Acquire preoperative CT images by CT scanning and save them as DICOM files, parse multiple frames of the DICOM files, and obtain a three-dimensional image matrix based on HU value expression; The HU values ​​of the three-dimensional image matrix are converted into pixel values, and the pixel values ​​are normalized and converted into a preset threshold range to obtain a three-dimensional image matrix corresponding to the preoperative CT image expressed based on the pixel values.

3. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 1 or 2, It is characterized in that The step of calculating the reference size data of the stone based on the kidney mask information includes: In the kidney mask area, multiple stone regions are detected by threshold segmentation method, and connected domain analysis is performed on each segmented stone region; The volume V of each stone area is calculated based on the voxels, and the stone diameter d is calculated according to the volume V; The volume V of each stone area and its corresponding stone diameter d are counted as stone reference size data.

4. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 3, It is characterized in that The method of acquiring intraoperative ultrasound video and performing adaptive optimization filtering algorithm preprocessing includes: Acquire the intraoperative ultrasound video and parse its corresponding AVI file, read the original frame rate data of the video frame, cut the intraoperative ultrasound video into single-frame ultrasound images according to the original frame rate data, and perform Z-score standardization and grayscale conversion on each frame of the ultrasound image; Using an adaptive optimized median filter algorithm to perform preliminary smoothing and noise reduction processing on the ultrasound image to remove salt and pepper noise and enhance the characteristics of the echo cluster in the ultrasound image; The ultrasound image is further smoothed and denoised using a Gaussian blur algorithm model to reduce the high-frequency noise of the ultrasound image.

5. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 4, It is characterized in that The method combines the NLM mean operation and the morphological operation to obtain the complete echo cluster in the intraoperative ultrasound video, including: The ultrasound image is processed using an NLM mean processing algorithm to retain ultrasound image details; The ultrasound image is processed by using a dilation operation and an erosion operation in morphological operations to remove noise between echo clusters and disconnected echo clusters, thereby retaining complete echo clusters in the ultrasound image.

6. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 5, It is characterized in that The method of screening out kidney stones from the complete echo group based on the stone reference size data and obtaining the real-time intraoperative coordinates of the kidney stones includes: By using a threshold segmentation method, a strong echo group is screened out from each complete echo group in the ultrasound image according to a preset strong echo group gray threshold; Performing connected domain analysis on each strong echo group in the ultrasound image, and calculating the area of ​​each strong echo group; The hyperechoic mass is equivalent to a circle, and the diameter of the hyperechoic mass is calculated according to the area of ​​the hyperechoic mass; Matching the diameter of each of the strong echo clusters with the stone diameter d in the stone reference size data; if the deviation between the diameter of the strong echo cluster and the stone diameter d is within the diameter tolerance range, then it is determined that the diameter is matched successfully, and the strong echo cluster with successfully matched diameter is determined to be a kidney stone; The center coordinates of the highly echogenic mass with successfully matched diameter in the ultrasound image are used as the real-time intraoperative coordinates of the kidney stone.

7. A kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images according to claim 6, It is characterized in that The method of screening out kidney stones from the complete echo group based on the stone reference size data and obtaining the real-time intraoperative coordinates of the kidney stones further includes: Acquire the coordinates of the contour points of the strong echo mass in the ultrasound image, and construct a three-dimensional model of the strong echo mass by combining the coordinates of the contour points of the strong echo mass in multiple frames of ultrasound images at multiple angles; Calculating the volume of the hyperechoic mass according to the three-dimensional model of the hyperechoic mass; Matching the volume of each of the strong echo clusters with the stone volume V in the stone reference size data; if the deviation between the volume of the strong echo cluster and the stone volume V is within the volume tolerance range, then the volume matching is determined to be successful; otherwise, the diameter of the strong echo cluster is recalculated; The center coordinates of the three-dimensional model of the strong echo group with successful volume matching are compared with the center coordinates of the strong echo group with successful diameter matching in the ultrasound image. If the difference between the two is within the coordinate tolerance range, the real-time intraoperative coordinates of the kidney stone are output; otherwise, the real-time intraoperative coordinates of the kidney stone are recalculated.

8. A kidney stone coordinate positioning system based on preoperative CT and intraoperative ultrasound images. It is characterized in that include: Data acquisition module, used to obtain preoperative CT images and intraoperative ultrasound videos; A data preprocessing module, used for analyzing the preoperative CT image and performing preprocessing, and for analyzing the intraoperative ultrasound video and performing preprocessing using an adaptive optimization filtering algorithm; A reference information acquisition module is used to extract kidney mask information from a pre-processed preoperative CT image using an improved nn-Unet model, and calculate stone reference size data based on the kidney mask information; the improved nn-Unet model includes a first-stage model and a second-stage model arranged in sequence, the first-stage model includes an encoder and a decoder, and the second-stage model includes a 3D U-net model; wherein: The encoder includes n layers of networks connected in sequence, n≥7, each layer of the network includes a first convolution module and a feature aggregation module, the convolution kernel size of the first convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network increases exponentially; the first convolution module is used to perform a convolution operation on the feature map input from the previous layer of the network to extract local feature information of the image; the feature aggregation module is used to downsample the feature map output by the first convolution module, perform a pooling operation, and obtain an aggregated feature map; The decoder includes n layers of networks connected in sequence and an output module, n≥7, each layer of the network includes a fully connected module, an upsampling module and a second convolution module, the convolution kernel size of the second convolution module is m×m, m≥7, and the number of convolution kernels in adjacent layers of the network decreases by half; the fully connected module is used to perform a residual connection between the feature map of the previous layer of the network and the output of the corresponding layer of the encoder; the upsampling module is used to perform a reverse convolution operation on the feature map output by the previous layer of the network, and upsample the image to obtain a prediction result with a higher resolution; the second convolution module is used to extract a reorganized high-order feature map from the output result of the upsampling module; the output module is used to output the high-order feature map; The second stage model is used to upsample the high-order feature map output by the first stage model to the size of the original image, and then input it into the 3D U-net model for segmentation at full resolution; An intraoperative stone extraction module is used to combine NLM mean operation and morphological operation to obtain a complete echo group in the intraoperative ultrasound video; based on the stone reference size data, kidney stones are screened out from the complete echo group to obtain the real-time intraoperative coordinates of the kidney stones; The output module is used to update the current position information of the kidney stone in real time and output the real-time intraoperative coordinates of the kidney stone.

9. An electronic device, It is characterized in that It includes a memory and a processor, and the processor is used to implement the steps of the kidney stone coordinate positioning method based on preoperative CT and intraoperative ultrasound images as described in any one of claims 1 to 7 when executing the computer management program stored in the memory.