An artificial intelligence-based kidney biopsy AI-assisted data processing method and system

Abstract

The application discloses an AI-assisted data processing method and system for kidney puncture based on artificial intelligence, and the method comprises the following steps: acquiring an ultrasound image containing a kidney and internal blood vessels of the kidney, performing noise suppression and contrast enhancement processing on the ultrasound image; performing blood vessel region segmentation on the internal blood vessels of the kidney in the ultrasound image, and placing the segmented blood vessel region in a predefined rectangular frame; correcting the blurred motion in the predefined rectangular frame caused by probe shaking; inputting the ultrasound image in the predefined rectangular frame into a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney; wherein the upper boundary line is located above the blood vessels and below the upper contour of the kidney; the lower boundary line separates the lower contour of the kidney from the blood vessels; and outputting the ultrasound image sequence to a kidney puncture assistance system for use by a doctor during the kidney puncture process. The application can accurately identify and extract the upper boundary line and the lower boundary line of the kidney, thereby improving the accuracy and safety of the puncture.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical image processing and artificial intelligence assisted diagnosis, in particular to a kidney puncture AI assisted data processing method and system based on artificial intelligence and a computing device. BACKGROUND

[0002] Kidney puncture is a common operation technique in the diagnosis of kidney diseases, which usually takes a small amount of kidney tissue with a biopsy needle under the guidance of ultrasound, and conducts detailed pathological examination and diagnosis. However, it is highly dependent on the experience and skills of doctors, especially in determining the puncture path and avoiding important vascular structures, and a slight mistake may lead to serious complications such as bleeding and hematoma formation.

[0003] Currently, kidney puncture usually relies on doctors to observe the kidney and its internal vascular structure in real time by holding an ultrasound probe, and manually selecting a puncture path. Since the selection of the puncture path is highly dependent on the experience and judgment of the doctor, there may be great differences between different doctors. And it is difficult to completely avoid hand shaking when holding the ultrasound probe, which affects the stability and clarity of the image. In addition, the doctor also needs to repeatedly adjust the puncture path in the real-time ultrasound image to ensure that important structures such as blood vessels are avoided, which increases the operation time and the risk of patients.

[0004] Although some image processing techniques (such as image filtering, enhancement, etc.) are applied to kidney puncture, the visual effect of the ultrasound image is improved. However, it is often limited to improving the image quality and cannot identify and label the upper and lower boundaries of the kidney. In addition, it also does not solve the problem of image blur or offset caused by ultrasound probe shaking, which further affects the selection of the puncture path.

[0005] To solve the above problems, the present application provides a kidney puncture AI assisted data processing method and system based on artificial intelligence to accurately identify and extract the upper and lower boundaries of the kidney, and thus improve the accuracy and safety of puncture. SUMMARY

[0006] In view of the above problems, the present application provides a kidney puncture AI assisted data processing method and system based on artificial intelligence and a computing device.

[0007] According to one aspect of the present application, a kidney puncture AI assisted data processing method based on artificial intelligence is provided, comprising:

[0008] acquiring an ultrasound image containing a kidney and its internal blood vessels, and performing noise suppression and contrast enhancement processing on the ultrasound image to improve the clarity of the puncture needle and blood vessel regions;

[0009] segmenting a blood vessel region of an internal blood vessel of a kidney in the ultrasound image, and placing the segmented blood vessel region in a predefined rectangular frame;

[0010] correcting a blurred motion in the predefined rectangular frame caused by a probe shaking;

[0011] inputting an ultrasound image in the predefined rectangular frame to a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing an upper boundary line and a lower boundary line of the kidney; wherein the upper boundary line is located above the blood vessel and below an upper contour of the kidney; and the lower boundary line separates a lower contour of the kidney from the blood vessel;

[0012] outputting the ultrasound image sequence to a kidney puncture assisting system for use by a doctor in a kidney puncture process.

[0013] In an alternative way, the noise suppression of the ultrasound image further comprises:

[0014] removing speckle and grain noise in the ultrasound image by a bilateral filtering algorithm, and outputting a weighted average pixel value of neighboring pixels; the weighted spatial value affinity factor is calculated according to spatial proximity and pixel value difference between pixels;

[0015] wherein the calculation formula of the spatial value affinity factor is:

[0016] ;

[0017] wherein, and are pixel coordinates; and are and corresponding pixel values; is a standard deviation of spatial proximity; is a standard deviation of pixel value difference.

[0018] In an alternative way, the blood vessel region segmentation of the internal blood vessel of the kidney in the ultrasound image further comprises:

[0019] selecting one or more pixel points located in the blood vessel as initial seed points, and checking neighboring pixels of the initial seed points;

[0020] if the similarity of the neighboring pixels to the initial seed points meets a preset threshold condition, adding the neighboring pixels to a current blood vessel region, and continuing to expand them as new seed points; repeat until no new neighboring pixels are added to the blood vessel region;

[0021] filling inner holes of the blood vessels according to a closing operation morphological algorithm, and smoothing boundaries of the blood vessels according to an opening operation morphological algorithm.

[0022] In an optional manner, the correcting the blurred motion caused by the probe shaking within the predefined rectangular frame further comprises:

[0023] detecting stable feature points in the continuous frames by an ORB feature point detection algorithm;

[0024] tracking the feature point motion of the stable feature points between the continuous frames by a Lucas-Kanade optical flow method;

[0025] estimating global motion parameters of the predefined rectangular frame based on the feature point motion;

[0026] applying an inverse transformation to a region within the predefined rectangular frame according to the estimated global motion parameters, so as to offset the blurred motion caused by the probe shaking.

[0027] In an optional manner, the kidney blood vessel segmentation model comprises an input layer, a feature extraction layer, a deformable convolution layer, an RPN region layer, a position regression layer, an RoI region selection layer, an SENet attention layer, a multi-scale feature fusion layer, and an output layer.

[0028] The deformable convolution layer is configured to be inserted into one or more layers of the feature extraction layer, so as to capture complex shape features of the blood vessels.

[0029] The RPN region layer is configured to generate a candidate region containing the kidney blood vessels and a boundary box thereof.

[0030] The position regression layer is configured to perform position regression on the candidate region, so as to adjust the boundary box thereof.

[0031] The RoI region selection layer is configured to extract a corresponding RoI region according to the adjusted boundary box of the position regression layer.

[0032] In an optional manner, the feature extraction layer comprises a ResNet50 residual network layer and an FPN feature network layer.

[0033] The deformable convolution layer is inserted into at least one layer of the feature extraction layer, and the deformable convolution layer comprises an offset prediction layer and a convolution operation layer with offset, wherein the offset prediction layer is configured to generate an offset map, and the convolution operation layer with offset is configured to generate a sampling network according to the offset map.

[0034] The RoI region selection layer comprises a mask layer, a coordinate extraction layer, and a classification layer; the mask layer is configured to generate a mask to filter out pixels containing blood vessels; the coordinate extraction layer is configured to extract the bounding box coordinates of the blood vessels; and the classification layer is configured to determine whether each RoI region contains blood vessels.

[0035] In an optional manner, the method further comprises:

[0036] According to the upper contour of the kidney, the blood vessels, the upper boundary line, and the lower boundary line in the sequence of ultrasound images, a puncture path is generated to avoid the key blood vessel region.

[0037] In an optional manner, the generating of the puncture path further comprises:

[0038] predicting the motion trajectory of the puncture needle under different puncture paths;

[0039] evaluating the safety of the puncture path according to the relationship between the motion trajectory and the blood vessel region;

[0040] maximizing the straightness of the puncture path according to the minimization of the contact probability between the puncture needle and the key blood vessel region;

[0041] superimposing the maximized puncture path on the sequence of ultrasound images in a visual manner for use by a doctor during a kidney puncture process.

[0042] According to another aspect of the present application, an artificial intelligence-based kidney puncture AI-assisted data processing system is provided, comprising:

[0043] a preprocessing module configured to acquire ultrasound images containing a kidney and internal blood vessels thereof, and to perform noise suppression and contrast enhancement processing on the ultrasound images to improve the clarity of the puncture needle and the blood vessel region;

[0044] a blood vessel region segmentation module configured to segment the internal blood vessels of the kidney in the ultrasound images and to place the segmented blood vessel region in a predefined rectangular frame;

[0045] a jitter elimination module configured to correct the blurred motion in the predefined rectangular frame caused by probe jitter;

[0046] a prediction module configured to input the ultrasound images in the predefined rectangular frame to a trained kidney blood vessel segmentation model to obtain a sequence of ultrasound images containing the upper boundary line and the lower boundary line of the kidney; the upper boundary line is located above the blood vessels and below the upper contour of the kidney; and the lower boundary line separates the lower contour of the kidney from the blood vessels;

[0047] an output module configured to output the sequence of ultrasound images to a kidney puncture assistance system for use by a doctor during a kidney puncture process.

[0048] According to another aspect of the present application, a computing device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface being in communication with each other through the communication bus.

[0049] According to the scheme provided by the present application, an ultrasound image containing a kidney and internal blood vessels thereof is acquired, noise suppression and contrast enhancement processing are performed on the ultrasound image to improve the clarity of the puncture needle and the blood vessel region; the internal blood vessels of the kidney in the ultrasound image are segmented into a blood vessel region, and the segmented blood vessel region is placed in a predefined rectangular frame; the blurred motion in the predefined rectangular frame caused by probe shaking is corrected; the ultrasound image in the predefined rectangular frame is input into a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney; wherein the upper boundary line is located above the blood vessel and below the upper contour of the kidney; the lower boundary line separates the lower contour of the kidney from the blood vessel; the ultrasound image sequence is output to a kidney puncture auxiliary system for use by a doctor during a kidney puncture process. The present application performs regional segmentation on the internal blood vessels of the kidney and places the segmentation result in a predefined rectangular frame, which not only reduces the tediousness of manual operation, but also improves the accuracy of segmentation. By correcting the blurred motion in the predefined rectangular frame, the image blurring problem caused by probe shaking is solved. By identifying and labeling the upper boundary line and the lower boundary line of the kidney through the kidney blood vessel segmentation model, not only the doctor is helped to quickly locate the puncture point, but also the puncture depth and direction are guided, thereby improving the accuracy and safety of puncture.

[0050] The above description is only a summary of the technical scheme of the present application. In order to enable the technical means of the present application to be more clearly understood, and to be implemented according to the content of the specification, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0051] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are intended to further aid the understanding of the preferred embodiments, and are not intended to limit the present application thereto. Moreover, like reference numerals designate like parts throughout the several views in the drawings. In the drawings:

[0052] Figure 1 A flowchart of an AI-based kidney puncture AI auxiliary data processing method according to an embodiment of the present application is shown;

[0053] Figure 2 A schematic diagram of a kidney blood vessel segmentation model according to an embodiment of the present application is shown;

[0054] Figure 3 A segmentation diagram of a kidney blood vessel of an embodiment of the present application is shown;

[0055] Figure 4 An ideal degree diagram of a needle of an embodiment of the present application is shown;

[0056] Figure 5 A framework diagram of an AI-assisted data processing system for kidney biopsy based on artificial intelligence of an embodiment of the present application is shown;

[0057] Figure 6 A structural diagram of a computing device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0058] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood and so that the scope of the present application can be conveyed to those skilled in the art.

[0059] Figure 1 A flow diagram of an AI-assisted data processing method for kidney biopsy based on artificial intelligence of an embodiment of the present application is shown. Specifically, as shown in Figure 1 the following steps are included:

[0060] Step S101, an ultrasound image containing a kidney and its internal blood vessels is acquired, and noise suppression and contrast enhancement processing are performed on the ultrasound image to improve the clarity of the regions of the puncture needle and blood vessels.

[0061] In this embodiment, through noise suppression and contrast enhancement, speckle noise and blurred edges in the image can be significantly reduced, especially the clarity of the regions of the puncture needle and blood vessels. Contrast enhancement processing can enlarge the gray difference between different tissues in the image, making the blood vessels and puncture needle more prominent in the image. This further helps doctors accurately identify the lesion area and provides real-time image guidance during the puncture operation.

[0062] Specifically, for noise suppression processing, multi-scale decomposition of the ultrasound image is performed using dyadic wavelet transform, and through setting a local threshold and combining soft threshold filtering method and hard threshold filtering method, the wavelet coefficients of different scales are shrunk to suppress speckle noise. Optionally, combined with morphological algorithms (such as hit or miss transform), the detailed information in the image is further extracted and retained, while the high-frequency noise below the threshold is removed.

[0063] For contrast enhancement processing, the extreme value stretching method of wavelet coefficients is used to enhance the contrast of the image. For example, by adjusting the dynamic range of the wavelet coefficients, the gray scale distribution of the image is more uniform, thereby improving the visual effect. Alternatively, by using an adaptive neighborhood histogram equalization method, the image is divided into high frequency and low frequency components, and the adaptive neighborhood histogram equalization algorithm is used to process the low frequency component to improve the overall contrast of the image; at the same time, the high frequency component is weighted to retain the detail information of the image.

[0064] In an alternative way, the noise suppression of the ultrasound image further comprises:

[0065] The speckle and grain noise in the ultrasound image is removed by a bilateral filtering algorithm, and a weighted average pixel value of the neighborhood pixel values is output; the weighted spatial value affinity factor is calculated according to the spatial proximity and pixel value difference between the pixels;

[0066] In this embodiment, the ultrasound image is suppressed by the bilateral filtering algorithm. Compared with the traditional filtering method (such as Gaussian filtering, median filtering, etc.), when calculating the new value of each pixel, not only the spatial proximity (i.e. distance) between the pixels is considered, but also the pixel value difference (i.e. brightness similarity) is considered. The pixels on both sides of the edge are not easily mixed in the filtering process, thereby maintaining the clear edge of the image. Moreover, for the speckle and grain noise commonly found in ultrasound images, the bilateral filtering can effectively remove these noises while avoiding excessive blurring of image details.

[0067] Specifically, a square filtering window is selected, which slides around each pixel to be processed. For each pixel in the window, the spatial proximity weight of the center pixel is calculated (the weight can be obtained by calculating the Euclidean distance between the two and mapping it to a weight range), the closer the distance, the greater the weight.

[0068] Similarly, for each pixel in the window, the pixel value difference weight of the center pixel is calculated (the weight can be obtained by calculating the difference between the two in the brightness space and mapping it to a weight range), the smaller the difference, the greater the weight.

[0069] The spatial proximity weight and the pixel value difference weight are multiplied to obtain the comprehensive weight of each neighborhood pixel. Then, the values of all pixels in the window are weighted and averaged by the comprehensive weight to obtain the new value of the center pixel. Each pixel in the image is traversed until the entire image is processed.

[0070] In this embodiment, the calculation formula of the spatial value affinity factor is:

[0071] ;

[0072] wherein, and are pixel coordinates; and are and corresponding pixel values; is a standard deviation of spatial proximity; is a standard deviation of color difference.

[0073] The above formula can flexibly control the relative importance of spatial proximity and color difference in the calculation of spatial value affinity factor, and can be applied to image processing tasks in different scenarios. Since the exponential function has strong robustness to noise and abnormal values, even if there are a few noises or abnormal pixel values, the influence can be weakened by adjusting the standard deviation parameter, thereby maintaining the stability of the overall spatial value affinity factor calculation.

[0074] Step S102, the blood vessels inside the kidney in the ultrasound image are segmented for blood vessel region, and the segmented blood vessel region is placed in a pre-defined rectangular frame.

[0075] By accurately segmenting the blood vessel region, the doctor can more easily identify the shape, size and position of the blood vessels, thereby improving the diagnostic accuracy of kidney diseases and reducing the risk of surgery.

[0076] In this embodiment, threshold-based, region growing or active contour model algorithms are used to segment the blood vessels inside the kidney to extract the blood vessel boundary. Optionally, morphological operations (erosion, dilation, opening operation, closing operation) are performed on the segmentation results to fill the holes.

[0077] According to the size and position of the segmented blood vessel region, one or more pre-defined rectangular frames are placed on the image, and the size and position of the rectangular frame can be adjusted according to actual needs to ensure that the blood vessel region is completely contained.

[0078] In an alternative way, the blood vessel region segmentation of the blood vessels inside the kidney in the ultrasound image further comprises:

[0079] Select one or more pixel points located in the blood vessel as initial seed points, and check the neighborhood pixels of the initial seed points;

[0080] If the similarity of the neighborhood pixels and the initial seed points meets the preset threshold condition, the neighborhood pixels are added to the current blood vessel region, and they are used as new seed points to continue to expand; repeat until no new neighborhood pixels are added to the blood vessel region;

[0081] According to the closing operation morphological algorithm, the internal holes of the blood vessels are filled, and according to the opening operation morphological algorithm, the boundaries of the blood vessels are smoothed.

[0082] In this embodiment, by selecting initial seed points and gradually expanding, the blood vessel region inside the kidney can be more accurately identified and segmented, especially in the case of blurred blood vessel boundaries. By automatically performing neighborhood pixel inspection, similarity comparison and region expansion through algorithms, manual intervention is reduced. Not only suitable for blood vessel segmentation of kidney ultrasound images, but also suitable for medical imaging fields such as CT, MRI, etc.

[0083] Specifically, one or more pixel points on the blood vessels inside the kidney are automatically selected as initial seed points. For example, possible blood vessel regions can be identified according to local features such as brightness, texture, etc. of the image, and the pixel points are automatically selected.

[0084] The neighborhood pixels (such as 8-neighborhood) of each initial seed point are inspected, and the similarity of the neighborhood pixels and the seed points in brightness or gradient features is calculated. If the similarity is greater than a preset threshold, the neighborhood pixel is added to the current blood vessel region and used as a new seed point to continue inspecting its neighborhood. Repeat the above process until no new neighborhood pixel meets the condition to join the blood vessel region.

[0085] Then, the closed operation is used to fill the possible holes inside the blood vessels. The closed operation is a process of first expansion and then corrosion, which can fill small holes without significantly increasing the boundary size.

[0086] Then, the open operation is used to smooth the blood vessel boundary and remove small protrusions or irregular boundaries caused by noise or other factors. The open operation is a process of first corrosion and then expansion, which can remove small objects smaller than the structure element.

[0087] Step S103, correcting the blurred motion in the pre-defined rectangular frame caused by probe shaking.

[0088] By eliminating the blur caused by probe shaking, the quality of the ultrasound image can be significantly improved, ensuring that the puncture needle accurately reaches the predetermined target point.

[0089] In an optional manner, the correction of the blurred motion in the pre-defined rectangular frame caused by probe shaking further comprises:

[0090] Detecting stable feature points in consecutive frames through an ORB feature point detection algorithm;

[0091] Tracking the feature point motion of the stable feature points between consecutive frames through a Lucas-Kanade optical flow method;

[0092] Estimating the global motion parameters of the pre-defined rectangular frame based on the feature point motion;

[0093] Applying an inverse transform to regions within the predefined rectangular box according to the estimated global motion parameters to counteract blurring motion due to probe jitter.

[0094] In this embodiment, the ORB (Oriented FAST and Rotated BRIEF) feature point detection algorithm combines the FAST key point detector and the BRIEF descriptor, and can quickly find the key points in the image. Among them, FAST and BRIEF are feature detection algorithm and vector creation algorithm respectively. ORB first searches for special areas from the image, called key points. The key point is a small area that protrudes in the image, such as a feature with a sharp change from light to dark pixel value. Then ORB calculates the corresponding feature vector for each key point. The feature of ORB is very fast, and to some extent, it is not affected by noise and image transformation, such as rotation and scaling transformation.

[0095] After detecting the feature points (key points), the Lucas-Kanade optical flow method is used to track the motion of the feature points between consecutive frames. The Lucas-Kanade optical flow method can calculate the displacement vector of the feature points between consecutive images, so as to track the motion vector of the feature points.

[0096] The traditional optical flow equation usually assumes constant brightness, that is, assumes that the brightness of the pixel point will not change between consecutive frames. The optical flow equation in this embodiment allows brightness changes because it is based on the comparison of pixel values within the window rather than the constant brightness assumption of a single pixel point. Even in the case of large changes in illumination, a more accurate optical flow estimate can be obtained. The expression of the optical flow equation is:

[0097]

[0098] Among them, is a preset size window; is the time interval between two frames of images; is the motion vector component of the pixel in the x and y directions; is a pixel position within the preset size window; is an image intensity function; is the image intensity at position and time . By using the weighted squared difference minimization strategy within the window, not only does it overcome the limitations of the traditional optical flow equation assuming constant brightness, but it also improves the accuracy of the optical flow estimate, especially in dealing with local noise and brightness changes, and performs better.

[0099] ​Based on the feature point-based motion vector, global motion parameters of the entire predefined rectangular frame can be estimated. According to the estimated global motion parameters, an inverse transform is applied to the region within the predefined rectangular frame, that is, a motion opposite to the probe jitter is performed to offset the blurred motion caused by the probe jitter.

[0100] In step S104, the ultrasound image within the predefined rectangular frame is input to the trained kidney vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney; the upper boundary line is located above the blood vessel and below the upper contour of the kidney; and the lower boundary line separates the lower contour of the kidney from the blood vessel.

[0101] In this embodiment, the kidney vessel segmentation model includes an input layer, a feature extraction layer, a deformable convolution layer, an RPN region layer, a position regression layer, an RoI region selection layer, an SENet attention layer, a multi-scale feature fusion layer, and an output layer, as shown in FIG. 1. Figure 2

[0102] The deformable convolution layer is inserted into one or more layers of the feature extraction layer to capture the complex shape features of the blood vessel. The conventional convolutional neural network is difficult to effectively capture the complex and variable shape features in the image when processing the image due to its fixed convolution kernel size, especially for the slender and curved structure such as the kidney vessel. The deformable convolution layer can more flexibly adapt to the change of the shape of the blood vessel by allowing the convolution kernel to dynamically sample the offset of the input feature map.

[0103] The RPN region layer is used to generate the candidate region containing the kidney vessel and the boundary box thereof. The conventional detection method is very time-consuming in extracting the candidate region, such as OpenCV adaboost using sliding window + image pyramid or R-CNN using SS (Selective Search). The RPN (Region Proposal Network) uses a series of CNN full convolution networks (the specific structure of which is not limited herein) to realize the sharing of the convolution features of the entire image with the detection network, thereby generating almost cost-free region recommendations, and the candidate region containing the kidney vessel and the boundary box thereof can be quickly generated.

[0104] The position regression layer is used to perform position regression on the candidate region to adjust the boundary box thereof to be closer to the real kidney vessel region.

[0105] The RoI region selection layer is used to extract the corresponding RoI region according to the boundary box adjusted by the position regression layer. After obtaining the feature maps through the convolution network, the RoI (Region of Interest) uses the selection search or RPN algorithm to obtain multiple target candidate boxes.​

[0106] In an alternative manner, the feature extraction layer comprises a ResNet50 residual network layer and an FPN feature network layer.

[0107] The deformable convolution layer inserts at least one deformable convolution layer in the feature extraction layer, the deformable convolution layer comprising an offset prediction layer and an offset convolution operation layer, wherein the offset prediction layer is used to generate an offset map; and the offset convolution operation layer is used to generate a sampling network according to the offset map.

[0108] The RoI region selection layer comprises a mask layer, a coordinate extraction layer and a classification layer; wherein the mask layer is used to generate a mask to filter out pixels containing blood vessels; the coordinate extraction layer is used to extract the boundary box coordinates of the blood vessels; and the classification layer is used to determine whether each RoI region contains blood vessels.

[0109] In this embodiment, by combining the ResNet50 residual network layer and the FPN (Feature Pyramid Network) feature network layer, rich feature information can be extracted at different scales, and different sizes of blood vessels in the blood vessel image can be effectively detected. The deformable convolution layer (Deformable Convolution) can enable the network to learn adaptive offsets during the sampling process, so as to more accurately locate the blood vessel boundary, especially for blood vessels with complex shape and high bending degree.

[0110] As shown in Figure 2 , the RoI (Region of Interest) region selection layer filters out the blood vessel region through the mask layer, further accurately extracts the blood vessel boundary box through the coordinate extraction layer, and finally the classification layer judges whether the RoI contains blood vessels, thereby realizing coarse-to-fine blood vessel detection. Among them, the mask layer uses a preset threshold or a mask generator learned to generate a mask of the blood vessel region from the feature map. The coordinate extraction layer uses an algorithm such as Canny edge detection to extract the boundary box coordinates of the blood vessels based on the mask region. After the classification layer extracts the features of each RoI region, it judges whether it contains blood vessels through a classifier (such as a fully connected layer, etc.).

[0111] Finally, the segmentation diagram of the kidney blood vessels as shown in Figure 3 is obtained, compared with the ideal degree diagram of the needle as shown in Figure 4 , the blood vessel region is placed in the specified rectangular frame, and the image is clear. The upper boundary line is above the blood vessels and below the upper contour of the kidney. The lower boundary line separates the lower contour of the kidney from the blood vessels (it should be noted that the blood vessels are not required to be completely above). In addition, the brightness information (i.e. clarity) of the needle is improved, and the influence of human factors such as probe shaking is reduced.

[0112] In an alternative way, the method further comprises:

[0113] Generating a puncture path according to the upper contour, blood vessels, upper boundary line and lower boundary line of the kidney in the sequence of ultrasound images, to avoid key blood vessel regions.

[0114] In an alternative way, the generating a puncture path further comprises:

[0115] Predicting a motion trajectory of the puncture needle under different puncture paths;

[0116] Evaluating the safety of the puncture path through the relationship between the motion trajectory and the blood vessel region;

[0117] Maximizing the straightness of the puncture path according to the minimization of the contact probability between the puncture needle and the key blood vessel region;

[0118] Superimposing the maximized puncture path on the sequence of ultrasound images in a visual manner for the doctor to use in the kidney puncture process.

[0119] In this embodiment, the trajectory equation of the puncture needle is:

[0120] ;

[0121] wherein, is the coordinate of the starting point of puncture, is the unit vector of the puncture direction, is the time parameter.

[0122] The safety evaluation needs to calculate the degree of overlap between the puncture trajectory and the blood vessel region. In this embodiment, the safety evaluation is approximated by calculating the minimum distance between the trajectory point and the blood vessel boundary, which is expressed as:

[0123] ;

[0124] wherein, is the time when the puncture ends; is a point on the blood vessel boundary, which is defined by the parameter .

[0125] In order to minimize the contact probability between the puncture needle and the key blood vessel region, and maximize the straightness of the puncture path, the following objective function is solved. The straightness is measured by the ratio of the length of the puncture path to the length of its straight segment projection, and the contact probability is estimated by the volume or area of the overlap between the path and the blood vessel region. In this embodiment, the weighted sum is used as the objective function, and the expression of the objective function is:

[0126] ;

[0127] wherein, , is a weight coefficient for balancing straightness and safety; Path Length refers to the total length of the path from the starting point to the ending point; Straight Line Projection refers to the length of the straight line segment from the starting point to the ending point, which is taken as the denominator, meaning that the first part of the entire expression tries to minimize the deviation of the path from the straight line distance, that is, to find a path as close to a straight line as possible; is an indicator function, when takes a value of 1 when located in the blood vessel region, otherwise 0; the integral term calculates the total time (or distance) of the puncture needle contacting the blood vessel region, for measuring the contact probability.

[0128] Finally, the calculated maximum puncture path (the path meeting the safety and straightness requirements) is superimposed on the ultrasound image sequence in a visual manner.

[0129] In step S105, the ultrasound image sequence is output to a kidney puncture auxiliary system for use by a doctor during a kidney puncture process.

[0130] According to the scheme provided by the present application, an ultrasound image containing a kidney and internal blood vessels thereof is acquired, noise suppression and contrast enhancement processing are performed on the ultrasound image to improve the clarity of the puncture needle and the blood vessel region, the internal blood vessels of the kidney in the ultrasound image are subjected to blood vessel region segmentation, and the segmented blood vessel region is placed in a predefined rectangular frame. The blurred motion in the predefined rectangular frame due to probe shaking is corrected. The ultrasound image in the predefined rectangular frame is input to a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney. The upper boundary line is located above the blood vessels and below the upper contour of the kidney. The lower boundary line separates the lower contour of the kidney from the blood vessels. The ultrasound image sequence is output to a kidney puncture auxiliary system for use by a doctor during a kidney puncture process. The present application performs region segmentation on the internal blood vessels of the kidney and places the segmentation result in a predefined rectangular frame, which not only reduces the tediousness of manual operation, but also improves the accuracy of segmentation. By correcting the blurred motion in the predefined rectangular frame, the image blurring problem caused by probe shaking is solved. By identifying and labeling the upper boundary line and the lower boundary line of the kidney through the kidney blood vessel segmentation model, the doctor is not only helped to quickly locate the puncture point, but also guided in the puncture depth and direction, thereby improving the accuracy and safety of the puncture.

[0131] Figure 5 The structure schematic diagram of the AI-assisted data processing system for kidney puncture based on artificial intelligence is shown. The AI-assisted data processing system for kidney puncture based on artificial intelligence comprises:

[0132] The preprocessing module 410 is used to acquire an ultrasound image containing the kidney and its internal blood vessels, and to perform noise suppression and contrast enhancement processing on the ultrasound image to improve the clarity of the puncture needle and blood vessel area.

[0133] The blood vessel region segmentation module 420 is used to segment the blood vessels inside the kidney in the ultrasound image and place the segmented blood vessel region within a predefined rectangle.

[0134] The jitter elimination module 430 is used to correct the blurry motion within the predefined rectangular frame caused by probe jitter.

[0135] The prediction module 440 is used to input the ultrasound image within the predefined rectangle into a trained renal vessel segmentation model to obtain an ultrasound image sequence containing the upper and lower boundaries of the kidney; wherein, the upper boundary is located above the vessel and below the upper contour of the kidney; the lower boundary separates the lower contour of the kidney from the vessel.

[0136] Output module 450 is used to output the ultrasound image sequence to the renal biopsy assist system for use by doctors during renal biopsy.

[0137] Figure 5 The diagram shows a structural schematic of an embodiment of the computing device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the computing device.

[0138] like Figure 5 As shown, the computing device may include: a processor 502, a communications interface 504, a memory 506, and a communications bus 508.

[0139] The processor 502, communication interface 504, and memory 506 communicate with each other via communication bus 508. Communication interface 504 is used to communicate with other network elements such as clients or other servers. The processor 502 executes program 510, specifically performing the relevant steps in the above-described embodiment of the AI-assisted data processing method for renal biopsy based on artificial intelligence.

[0140] Specifically, program 510 may include program code that includes computer operation instructions.

[0141] The processor 502 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application. The one or more processors included in the computing device can be of the same type or different types, such as one or more CPUs and one or more ASICs.

[0142] The memory 506 stores the program 510. The memory 506 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory.

[0143] According to the scheme provided by the present application, an ultrasound image containing a kidney and internal blood vessels thereof is acquired, noise suppression and contrast enhancement processing are performed on the ultrasound image to improve the clarity of the puncture needle and blood vessel region; the internal blood vessels of the kidney in the ultrasound image are segmented into blood vessel regions, and the segmented blood vessel regions are placed in a predefined rectangular frame; the blurred motion in the predefined rectangular frame caused by probe shaking is corrected; the ultrasound image in the predefined rectangular frame is input into a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney; the upper boundary line is located above the blood vessels and below the upper contour of the kidney; the lower boundary line separates the lower contour of the kidney from the blood vessels; the ultrasound image sequence is output to a kidney puncture auxiliary system for use by a doctor during a kidney puncture process. The present application performs regional segmentation on the internal blood vessels of the kidney and places the segmentation results in a predefined rectangular frame, which not only reduces the tediousness of manual operation, but also improves the accuracy of segmentation. By correcting the blurred motion in the predefined rectangular frame, the image blurring problem caused by probe shaking is solved. By identifying and labeling the upper boundary line and the lower boundary line of the kidney through the kidney blood vessel segmentation model, the doctor is not only helped to quickly locate the puncture point, but also guided in the puncture depth and direction, thereby improving the accuracy and safety of the puncture.

[0144] Those skilled in the art will appreciate that the modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and furthermore can be divided into multiple sub-modules or sub-units or sub-components. Any combination of all the features disclosed in the present specification (including the accompanying claims, abstract and drawings), and any method or apparatus so disclosed, can be taken in any combination, except that at least some of such features and / or processes or units are mutually exclusive, unless explicitly stated otherwise. Each feature disclosed in the present specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features serving the same, equivalent or similar purpose, unless explicitly stated otherwise. Furthermore, the skilled person will appreciate that the combination of features of different embodiments implies that the features of the different embodiments are meant to be combined, unless explicitly stated otherwise. For example, in the claims below, any of the embodiments can be used in any combination. The application can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In the unitary claim, several of the devices mentioned in the embodiments can be implemented by means of one and the same hardware item. The steps of the above-described embodiments, unless explicitly stated otherwise, are not to be understood as having to be carried out in the order in which they are described.

Claims

1. An artificial intelligence-based kidney puncture AI-assisted data processing method, characterized in that, The method comprises the following steps: acquiring an ultrasound image containing a kidney and internal blood vessels thereof, and performing noise suppression and contrast enhancement on the ultrasound image to improve the clarity of the puncture needle and the blood vessel region; segmenting the internal blood vessels of the kidney in the ultrasound image to obtain a segmented blood vessel region, and placing the segmented blood vessel region in a predefined rectangular frame; correcting the blurred motion caused by probe jitter in the predefined rectangular frame; inputting the ultrasound image in the predefined rectangular frame into a trained kidney blood vessel segmentation model to obtain an ultrasound image sequence containing the upper boundary line and the lower boundary line of the kidney; the upper boundary line is located above the blood vessels and below the upper contour of the kidney; the lower boundary line separates the lower contour of the kidney from the blood vessels; outputting the ultrasound image sequence to a kidney puncture auxiliary system for use by a doctor during a kidney puncture process; the noise suppression on the ultrasound image further comprises the following steps: The speckle and grain noise in the ultrasound image is removed by a bilateral filtering algorithm, and a weighted average pixel value of neighboring pixels is output; the weight of the weighted value is calculated according to the spatial proximity and pixel value difference between pixels; the calculation formula of the weight is: wherein, and is a pixel coordinate; and is and is a corresponding pixel value; is a standard deviation of spatial proximity; is a standard deviation of pixel value difference; and the step of correcting the blurring motion caused by the probe shaking in the predefined rectangular frame further comprises: detecting stable feature points in consecutive frames by an ORB feature point detection algorithm; tracking the feature point motion of the stable feature points between consecutive frames by a Lucas-Kanade optical flow method; estimating the global motion parameters of the predefined rectangular frame based on the feature point motion; applying an inverse transform to the region in the predefined rectangular frame according to the estimated global motion parameters to offset the blurred motion caused by probe jitter; the expression of the optical flow equation is: wherein, is a preset size window; is a time interval between two frames of images; is a motion vector component of a pixel in x and y directions; is a pixel position within the preset size window; is an image intensity function; is an image intensity at position and time t. 2.The AI-based kidney puncture AI assistance data processing method of claim 1, wherein, the blood vessel region segmentation of the internal blood vessels of the kidney in the ultrasound image further comprises the following steps: selecting one or more pixel points located in the blood vessels as initial seed points, and checking the neighborhood pixels of the initial seed points; if the similarity of the neighborhood pixels to the initial seed points meets a preset threshold condition, the neighborhood pixels are added to the current blood vessel region and used as new seed points for further expansion; the process is repeated until no new neighborhood pixels are added to the blood vessel region; filling the internal holes of the blood vessels according to a closing morphological algorithm, and smoothing the boundaries of the blood vessels according to an opening morphological algorithm.

3. The AI-based kidney puncture AI auxiliary data processing method according to claim 2, wherein: the kidney blood vessel segmentation model comprises an input layer, a feature extraction layer, a deformable convolution layer, an RPN region layer, a position regression layer, an RoI region selection layer, an SE-Net attention layer, a multi-scale feature fusion layer, and an output layer; the deformable convolution layer is inserted into one or more layers of the feature extraction layer to capture the complex shape features of the blood vessels; the RPN region layer is used to generate candidate regions containing kidney blood vessels and their bounding boxes; the position regression layer is used to perform position regression on the candidate regions to adjust their bounding boxes; the RoI region selection layer is used to extract the corresponding RoI region according to the bounding boxes adjusted by the position regression layer.

4. The AI-based kidney puncture AI auxiliary data processing method according to claim 3, wherein: the feature extraction layer comprises a ResNet50 residual network layer and an FPN feature network layer; The deformable convolution layer inserts at least one deformable convolution layer in the feature extraction layer, and the deformable convolution layer comprises an offset prediction layer and a convolution operation layer with offset, wherein the offset prediction layer is used to generate an offset map; and the convolution operation layer with offset is used to generate a sampling network according to the offset map. The RoI region selection layer comprises a mask layer, a coordinate extraction layer and a classification layer; wherein the mask layer is used to generate a mask to screen out pixels containing blood vessels; the coordinate extraction layer is used to extract the boundary box coordinates of the blood vessels; and the classification layer is used to determine whether each RoI region contains blood vessels. 5.The AI-based kidney puncture AI assistance data processing method of claim 4, wherein, The method further comprises: According to the upper contour of the kidney, the blood vessels, the upper boundary line and the lower boundary line in the sequence of ultrasound images, a puncture path is generated to avoid the key blood vessel region. 6.The AI-based kidney puncture AI assistance data processing method of claim 5, wherein, The generation of the puncture path further comprises: Predicting the motion trajectory of the puncture needle under different puncture paths; Evaluating the safety of the puncture path through the relationship between the motion trajectory and the blood vessel region; Maximizing the straightness of the puncture path according to the minimization of the contact probability between the puncture needle and the key blood vessel region; Superimposing the maximized puncture path on the sequence of ultrasound images in a visual manner for use by a doctor during the kidney puncture process.

7. An artificial intelligence-based kidney puncture AI-assisted data processing system, characterized by, The system is used to perform the method of any one of claims 1 to 6, comprising: A preprocessing module is configured to acquire ultrasound images containing a kidney and internal blood vessels thereof, and perform noise suppression and contrast enhancement processing on the ultrasound images to improve the clarity of the puncture needle and the blood vessel region; A blood vessel region segmentation module is configured to perform blood vessel region segmentation on the internal blood vessels of the kidney in the ultrasound images, and place the segmented blood vessel region in a predefined rectangular frame; A jitter elimination module is configured to correct the blurred motion in the predefined rectangular frame caused by probe jitter; A prediction module is configured to input the ultrasound images in the predefined rectangular frame to a trained kidney blood vessel segmentation model to obtain a sequence of ultrasound images containing the upper boundary line and the lower boundary line of the kidney; wherein the upper boundary line is located above the blood vessels and below the upper contour of the kidney; and the lower boundary line separates the lower contour of the kidney from the blood vessels; An output module is configured to output the sequence of ultrasound images to a kidney puncture assistance system for use by a doctor during the kidney puncture process.

8. A computing device comprising: A processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute the method of any one of claims 1 to 6.

Images