An intraoperative robotic imaging analysis system for renal tumor resection

Abstract

The present invention relates to the field of image recognition technology, and particularly relates to an intraoperative robot imaging analysis system for renal tumor resection. The system obtains ultrasonic images during renal tumor resection, forms a training set with partial ultrasonic images, equally divides the ultrasonic images into several ultrasonic blocks. Among them, each ultrasonic block corresponds to an ultrasonic echo signal, obtains the energy of the ultrasonic echo signal under different frequency components, thereby determines the suppression amount of the ultrasonic echo signal, uses it to screen out ultrasonic blocks with high suppression amounts, then determines the tumor tissue heterogeneity of the ultrasonic blocks with high suppression amounts, uses it to obtain the upsampling weights of pixel points in the ultrasonic image, thereby trains the training set to obtain a trained semantic segmentation network model, and uses it to segment the renal tumor area in the ultrasonic image. The present invention improves the accuracy of renal tumor area recognition through adaptive upsampling weights, thereby ensuring the effect of renal tumor resection.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly relates to an intraoperative robot imaging analysis system for renal tumor resection. Background Art

[0002] In renal tumor resection surgery, the application of an intraoperative robot imaging analysis system plays a very important role, which can greatly improve the accuracy and safety of the surgery. Its system composition and structure include: (1) Surgeon's console: The core of the robotic surgery system, through which the doctor performs surgical operations. The doctor can view the image data in real time through the console and control the robotic arm to perform precise resection operations. The console needs to be equipped with a high-resolution display and precise control equipment to have a clearer understanding of the situation of the kidney tumor and surrounding tissues. (2) Bedside robotic arm system: The accuracy and flexibility of the robotic arm enable the doctor to more easily locate the tumor, excise tissues, and stop bleeding during the operation, thereby reducing the surgical risk. (3) Imaging system: The imaging system is a key component of this robotic imaging analysis system. It can obtain real-time renal imaging information of the patient, provide high-resolution two-dimensional or three-dimensional images, and help the doctor more accurately locate the tumor, renal structure, and surrounding important tissues. The imaging system usually uses imaging technologies such as CT, MRI, or ultrasound, and is combined with the robotic system in real time for image fusion.

[0003] Existing problems: In renal tumor resection surgery, accurately identifying and extracting the tumor area is crucial. However, since tumors are often located inside the kidney and are often relatively hidden or grow in other tissues, solely relying on robotic images to determine the boundary and extent of the tumor will cause a certain degree of error due to the blurred edge information of the tumor, resulting in a reduced accuracy in the machine-assisted recognition process. Summary of the Invention

[0004] The present invention provides an intraoperative robot imaging analysis system for renal tumor resection to solve the existing problems.

[0005] The following technical solutions are adopted for an intraoperative robot imaging analysis system for renal tumor resection of the present invention:

[0006] An embodiment of the present invention provides an intraoperative robot imaging analysis system for renal tumor resection, and the system includes the following modules:

[0007] Renal tumor resection image acquisition module: used to obtain several frames of ultrasonic images during renal tumor resection, and form a training set with some of the ultrasonic images; equally divide each frame of ultrasonic image into several ultrasonic blocks; wherein, each ultrasonic block corresponds to an ultrasonic echo signal;

[0008] High suppression amount ultrasonic block recognition module: used to obtain the energy of each ultrasonic echo signal at different frequency components; determine the suppression amount of each ultrasonic echo signal according to the magnitude of the energy at different frequency components and the extreme value difference in the ultrasonic echo signal; screen out high suppression amount ultrasonic blocks according to the magnitude of the suppression amount;

[0009] Tumor tissue heterogeneity analysis module: used to determine the tumor visibility of each high suppression amount ultrasonic block according to the pixel gray value difference between each high suppression amount ultrasonic block and adjacent ultrasonic blocks; determine the tumor tissue heterogeneity of each high suppression amount ultrasonic block according to the tumor visibility and the magnitude of the suppression amount of each high suppression amount ultrasonic block;

[0010] Renal tumor region recognition module: used to determine the upsampling weight of each pixel point in the ultrasonic image according to the magnitude of the tumor tissue heterogeneity; train the training set according to the upsampling weight to obtain a trained semantic segmentation network model; use the trained semantic segmentation network model to segment the renal tumor region in the ultrasonic image.

[0011] Further, the determination of the suppression amount of each ultrasonic echo signal includes:

[0012] Divide each ultrasonic echo signal into several ultrasonic echo signal segments;

[0013] Calculate the mean value of all maximum values in each ultrasonic echo signal segment as the amplitude of each ultrasonic echo signal segment;

[0014] Determine the ultrasonic variability of each ultrasonic echo signal according to the difference between the amplitudes of the ultrasonic echo signal segments in each ultrasonic echo signal;

[0015] Determine the ultrasonic regularity of each ultrasonic echo signal according to the magnitude of the energy of each ultrasonic echo signal at different frequency components;

[0016] Determine the suppression amount of each ultrasonic echo signal according to the ultrasonic variability and ultrasonic regularity of each ultrasonic echo signal.

[0017] Further, the determination of the ultrasonic variability of each ultrasonic echo signal includes:

[0018] In the amplitudes of all ultrasonic echo signal segments of each ultrasonic echo signal, take the difference between the maximum amplitude and the minimum amplitude as the ultrasonic variability of each ultrasonic echo signal.

[0019] Further, the determination of the ultrasonic regularity of each ultrasonic echo signal includes:

[0020] For any ultrasonic echo signal, obtain the maximum value among the energies at all different frequency components, and then obtain the sum value of the energies at all different frequency components. Denote the ratio of the maximum value to the sum value as the ultrasonic regularity of the any ultrasonic echo signal.

[0021] Further, the determining the suppression amount for each ultrasonic echo signal according to the ultrasonic variability and ultrasonic regularity of each ultrasonic echo signal includes:

[0022] Use the normalized value of the ratio of the ultrasonic variability to the ultrasonic regularity of each ultrasonic echo signal as the suppression amount for each ultrasonic echo signal.

[0023] Further, the screening out of the high-suppression-amount ultrasonic blocks includes:

[0024] Denote the ultrasonic blocks corresponding to the ultrasonic echo signals with the suppression amount greater than or equal to the preset suppression amount threshold as the high-suppression-amount ultrasonic blocks.

[0025] Further, the determining the tumor visibility of each high-suppression-amount ultrasonic block includes:

[0026] Calculate the mean value of the gray values of all pixel points within each ultrasonic block as the gray mean value of each ultrasonic block;

[0027] Calculate the mean value of the gray values of all pixel points in each frame of ultrasonic image as the gray mean value of each frame of ultrasonic image;

[0028] Determine the tumor visibility of each high-suppression-amount ultrasonic block according to the gray mean values of each frame of ultrasonic image and each ultrasonic block.

[0029] Further, the determining the tumor visibility of each high-suppression-amount ultrasonic block according to the gray mean values of each frame of ultrasonic image and each ultrasonic block includes:

[0030] For any high-suppression-amount ultrasonic block, denote the ratio of the gray mean value of the ultrasonic image where the high-suppression-amount ultrasonic block is located to the gray mean value of the high-suppression-amount ultrasonic block as the first ratio, denote the sum value of the absolute values of the differences between the gray mean value of the high-suppression-amount ultrasonic block and the gray mean values of all adjacent ultrasonic blocks as the first sum value, and denote the normalized value of the ratio of the first ratio to the first sum value as the tumor visibility of the any high-suppression-amount ultrasonic block.

[0031] Further, the determining the tumor tissue heterogeneity of each high-suppression-amount ultrasonic block includes:

[0032] For any high suppression amount ultrasound block, the product of the suppression amount of the high suppression amount ultrasound block and a preset first allocation weight is denoted as the first product, the product of the tumor visibility of the high suppression amount ultrasound block and a preset second allocation weight is denoted as the second product, and the sum value of the first product and the second product is used as the tumor tissue heterogeneity of any one of the high suppression amount ultrasound blocks; wherein, the sum value of the preset first allocation weight and the preset second allocation weight is 1.

[0033] Further, the determination of the upsampling weight of each pixel point in the ultrasound image includes:

[0034] Set the upsampling weight of each pixel point in any high suppression amount ultrasound block to be the sum value of a preset basic weight and the tumor tissue heterogeneity of any one of the high suppression amount ultrasound blocks;

[0035] Set the upsampling weight of each pixel point not in the high suppression amount ultrasound block to be the preset basic weight.

[0036] The beneficial effects of the technical solution of the present invention are:

[0037] In the embodiment of the present invention, an ultrasound image during renal tumor resection is obtained, a partial ultrasound image is used as a training set, the ultrasound image is equally divided into several ultrasound blocks, wherein each ultrasound block corresponds to an ultrasound echo signal, the energy of the ultrasound echo signal at different frequency components is obtained, so as to determine the suppression amount of the ultrasound echo signal, and high suppression amount ultrasound blocks are screened out. Then, the tumor tissue heterogeneity of the high suppression amount ultrasound blocks is determined to obtain the upsampling weight of the pixel points in the ultrasound image. Thus, by combining the ultrasound echo and the local tissue pixel structure for analysis, the position of the renal tumor is accurately determined, relatively comprehensive and reliable tumor distribution information is obtained, and the upsampling weight is adaptively adjusted to improve the training pertinence, ensuring the recognition robustness and accuracy of the network. Finally, the training set is trained to obtain a trained semantic segmentation network model for segmenting the renal tumor area in the ultrasound image. Thus, the present invention improves the accuracy of renal tumor area recognition through adaptive upsampling weight, thereby ensuring the effect of renal tumor resection. Description of the Drawings

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 It is a module flowchart of an intraoperative robot image analysis system for renal tumor resection according to the present invention;

[0040] Figure 2 It is a schematic diagram of the change in ultrasonic echo energy;

[0041] Figure 3 It is a frequency-domain diagram of the Fourier transform of the ultrasonic echo signal;

[0042] Figure 4 It is a schematic diagram outlining the optimization training process for renal tumor recognition. Detailed implementation manners

[0043] In order to further elaborate on the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation manners, structures, features, and effects of an intraoperative robot imaging analysis system for renal tumor resection proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. In addition, the specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field to which the present invention belongs.

[0045] The following specifically describes the specific solution of an intraoperative robot imaging analysis system for renal tumor resection provided by the present invention in conjunction with the accompanying drawings.

[0046] Please refer to Figure 1 , which shows a module flowchart of an intraoperative robot imaging analysis system for renal tumor resection provided by an embodiment of the present invention. The system includes the following modules:

[0047] Module 101: Renal tumor resection image acquisition module.

[0048] This module is used to obtain a number of ultrasonic images during renal tumor resection, and form a training set with some of the ultrasonic images; each ultrasonic image is equally divided into a number of ultrasonic blocks; among them, each ultrasonic block corresponds to an ultrasonic echo signal.

[0049] It should be noted that during the operation of renal tumor resection, when analyzing and extracting the edges of renal tumors through image recognition (semantic segmentation network), since tumor tissues are often partially or completely hidden within the kidney, it is difficult for traditional image processing methods to accurately identify the actual location and boundaries of tumors, resulting in inaccurate segmentation results, thereby affecting the doctor's tumor resection plan. In this embodiment, by configuring and preprocessing intraoperative tumor images and their ultrasound data as an image analysis data set, obtaining the configured data set, analyzing the ultrasound suppression amount of the images, and combining the heterogeneity of tumor tissues in the high suppression amount area, the semantic training upsampling weight value is determined, and then the optimized training result is obtained. Based on the optimized parameters, the intraoperative tumor area is identified and extracted, improving the accuracy and efficiency of the doctor's tumor resection.

[0050] Obtain several frames of ultrasound images during renal tumor resection. Use some of the ultrasound images to form a training set. Divide each frame of ultrasound image into several ultrasound blocks, where each ultrasound block corresponds to an ultrasonic echo signal.

[0051] It should be noted that in this embodiment, the ultrasound image is a grayscale image. The specific process of configuring and preprocessing intraoperative tumor images and their ultrasound data as an image analysis data set is as follows: (1) Image data configuration: First, use intraoperative imaging devices (i.e., intraoperative ultrasound, CT, and MRI, etc.) to obtain the original images of the tumor site, that is, ultrasound images. During the acquisition process, it is necessary to ensure that the images have sufficient resolution, clarity, and integrity. Classify and store the images according to information such as the patient, site, and time, that is, through naming standardization, establish a unified data directory structure (for example: PatientID / Modality / Timepoint). Convert the original image format (DICOM) to the standard image format (PNG) supported by the model (semantic segmentation model), and retain key metadata (including but not limited to pixel spacing, slice thickness, and imaging parameters, etc.) for subsequent registration and three-dimensional reconstruction. (2) Ultrasound data configuration: In the robotic imaging system, for the generated several frames of ultrasound images, in addition to their own pixel information as the training result, in order to achieve accurate identification and judgment of renal tumors under multiple shapes, it is also necessary to record the ultrasonic echo characteristics of each pixel area as a preset condition during the training process, that is, set For the local area, it is necessary to ensure that the ultrasound image can be evenly divided. Taking this as an example, each frame of ultrasound image is divided into several Ultrasonic blocks, and obtain the ultrasonic echo signals of each ultrasonic block in each frame of ultrasonic image during the process of machine recognition or resection through an ultrasonic device. (3) Ultrasonic image preprocessing: Remove the artifacts and noise of the intraoperative ultrasonic image through an image denoising algorithm (Gaussian filtering algorithm), then perform enhancement processing on the ultrasonic image (image contrast stretching algorithm) to improve the visibility of the tumor boundary, and then normalize the pixel values in the ultrasonic image (0-1 standardization method) to unify the model input. Among them, the Gaussian filtering algorithm, the image contrast stretching algorithm, and the 0-1 standardization method are all well-known technologies, and the specific methods will not be introduced here. And to expand the sample size and improve the robustness of the model, perform data augmentation operations such as rotation, flipping, scaling, and noise addition on the tumor ultrasonic image. (4) Dataset construction and formatting: Randomly divide the preprocessed image data into a training set, a validation set, and a test set. The ratio of the number of ultrasonic images in the three datasets is 8:1:1. This is used as an example for description. Among them, the training set is the main source for the neural network to learn and fit data. The validation set is used to monitor the performance of the model during the training process, help adjust hyperparameters, and prevent overfitting. The test set is used to finally evaluate the generalization ability of the model. Package the image and labels (such as: patient, acquisition site, and acquisition time, etc.) into a TFRecord array. This array is a format supported by deep learning. The images in this array format have the basic kidney tumor characteristics of the original images and will be applied to subsequent semantic segmentation training. This is a well-known technology.

[0052] Module 102: High suppression amount ultrasonic block recognition module.

[0053] This module is used to obtain the energy of each ultrasonic echo signal at different frequency components; determine the suppression amount of each ultrasonic echo signal according to the magnitude of the energy at different frequency components and the extreme value difference in the ultrasonic echo signal; and screen out the ultrasonic blocks with high suppression amounts according to the magnitude of the suppression amount.

[0054] It should be noted that: During the resection of kidney tumors, in order to achieve the purpose of assisting the robot in resection, in the imaging system, a corresponding recognition module is equipped. This recognition module is a semantic segmentation network that has the function of recognizing and segmenting kidney tumors after training. Generally, the semantic segmentation network cannot achieve the purpose of recognizing kidney tumors in special cases due to limited training process. In some images, kidney tumors show a hidden growth state, that is, they grow inward, and no relevant tumor traces or edges can be directly observed on the kidney surface. To solve this problem, during the training process, it is necessary to optimize and focus on annotating the training of relevant datasets to improve the network's recognition, analysis, and perception ability for multi-morphological kidney tumors. Therefore, it is necessary to train the information of kidney tumors in different dimensions (ultrasonic echo dimension and image dimension).

[0055] The information contained in each frame of ultrasound image in the dataset to be trained is: the pixel information of the image itself after preprocessing and the ultrasound echo information corresponding to each ultrasound block. A schematic diagram of the change in ultrasound echo energy is shown in Figure 2 as shown Figure 2 In the ultrasound echo signal in, the horizontal axis is the ultrasound echo time and the vertical axis is the ultrasound echo energy. Figure 2 For the first half part with a higher amplitude in the ultrasound echo signal of, it is the area of other normal tissues in the kidney. For the second half part with a lower amplitude and more noise, since ultrasound has a higher penetrability for the kidney tumor area, the echo energy is relatively low. At this time, the probability of being in the kidney tumor area increases significantly (even if it is not visible visually). The ultrasound information and pixel information in the image greatly reflect the amount of tumor information contained in the image. In order to make up for the problem of poor training effect of the semantic segmentation network caused by the unclear features such as the growth shape of the tumor, it is necessary to perform relevant recognition and analysis on the training set, and optimize the training process according to the recognition results, so that the network has a high-precision tumor recognition ability during actual intraoperative surgery, that is, taking the ultrasound information in each image as a training optimization reference index.

[0056] It should be further noted that: the characteristics of the change in ultrasound signals can reflect the local tumor characteristics during the process of the robot collecting training images. Many types of tumors, especially tumors rich in liquid components (such as cystic tumors), usually present the characteristic of low echo. The specific reason is that liquid or semi-liquid tumor tissues have weak reflection of ultrasonic waves, and the sound waves can penetrate deeper, so the echo is weak, forming a darker area. When the tumor contains more calcifications, fibrous tissues or vascular structures, such parts usually show high echo. The specific reason is that the density of such components is relatively large, which leads to more ultrasonic waves being reflected back, forming a bright white echo area. Thus, the ultrasound echo curve information carried by each ultrasound block in the image to be trained can be obtained, and the abnormal change of tumor ultrasound energy can be analyzed. When the energy of the echo curve changes greatly in amplitude, its corresponding abnormal change is higher, that is, the renal tissue reflected by the ultrasound echo is more abnormal.

[0057] Preferably, in an embodiment of the present invention, the method for obtaining high-inhibition ultrasound blocks includes:

[0058] For the ultrasound echo signal corresponding to any ultrasound block in any frame of ultrasound image, the ultrasound echo signal is equally divided into several ultrasound echo signal segments.

[0059] It should be noted that: in this embodiment, the preset number of equal segments is 10, and this is taken as an example for description. That is, according to time, each ultrasound echo signal is equally divided into 10 segments. If the total length of the time dimension occupied by the ultrasound echo signal is 10 seconds, then the length of each ultrasound echo signal segment occupied by the time dimension is 1 second.

[0060] Using the first derivative method, several maximum values in each ultrasonic echo signal segment are obtained.

[0061] Among them, the first derivative method is a well-known technology, and the specific method will not be introduced here.

[0062] Calculate the mean value of all maximum values in each ultrasonic echo signal segment as the amplitude of each ultrasonic echo signal segment.

[0063] Among the amplitudes of all ultrasonic echo signal segments of any one ultrasonic echo signal, the difference between the maximum amplitude and the minimum amplitude is used as the ultrasonic variability of the any one ultrasonic echo signal.

[0064] It should be noted that: when the mean values of the maximum values of different ultrasonic echo signal segments in the ultrasonic block fluctuate greatly, it proves that there is partial interval abnormality in the ultrasonic echo of this block, and then the variability of the ultrasonic block is higher at this time. That is, the larger this difference is, the higher the ultrasonic variability is, which indicates that after passing through the ultrasonic wave, compared with the normal renal tissue area, some local areas can show the characteristics of low reflection and high absorption (low echo area) or high reflection and low absorption (high echo area) for the ultrasonic wave, and then the possibility of the existence of tumors in the corresponding such areas is higher.

[0065] It should be further noted that: since the echo of tumor tissue is usually relatively irregular, this is mainly because of the heterogeneity of the cell structure and blood vessels and connective tissue inside the tumor tissue. Compared with normal kidney tissue, tumor tissue often shows heterogeneity, and the echo shows uneven, fragmented or strongly discontinuous echo information. This irregularity usually comes from the mixture of different tissue components in the tumor. Therefore, compared with the normal tissue area, the ultrasonic echo periodicity of the renal tumor area is often lower, or there is no corresponding regular period. Then, there will be a large difference between the two types of areas in the frequency domain. For the normal tissue area of the kidney, its ultrasonic echo signal shows stable and regular changes, the signal amplitude is relatively fixed, and the corresponding frequency domain information is relatively concentrated. On the contrary, for the renal tumor area, due to the variability of the ultrasonic echo signal, the frequency information is relatively dispersed in the frequency domain.

[0066] Perform short-time Fourier transform on any one ultrasonic echo signal to obtain the energies under several different frequency components.

[0067] Among them, the short-time Fourier transform is a well-known technology, and the specific method will not be introduced here. The Fourier transform frequency domain diagram of the ultrasonic echo signal is as Figure 3 shown, Figure 3 where the horizontal axis is the frequency, the unit is Hertz (Hz), and the vertical axis is the frequency amplitude, Figure 3 and the data points in the circle correspond to 2.0 Hertz (Hz).

[0068] For any ultrasonic echo signal, obtain the maximum value of the energies at all different frequency components, and then obtain the sum value of the energies at all different frequency components. Denote the ratio of the maximum value to the sum value as the ultrasonic regularity of the any ultrasonic echo signal.

[0069] It should be noted that: By analyzing the proportion of the maximum frequency-domain energy to the total energy in the frequency domain, the regularity of the ultrasonic block echo signal can be determined. The higher this proportion, that is, the closer it is to 1, the higher the energy of a certain frequency domain and the more concentrated the energy distribution. Then for the time-domain signal, its periodicity is higher, that is, the closer it is to the normal tissue area of the kidney, because through the above analysis, the echo of the tissue area is relatively regular. On the contrary, when this proportion is lower, that is, the closer it is to 0, the more dispersed the energy distribution in the corresponding frequency domain, and the more similar it is to the tumor area of the kidney in characteristics.

[0070] Take the normalized value of the ratio of the ultrasonic variability to the ultrasonic regularity of any ultrasonic echo signal as the suppression amount of the any ultrasonic echo signal.

[0071] It should be noted that: In this embodiment, use a linear normalization function to normalize the ratio of the ultrasonic variability to the ultrasonic regularity to the interval. Take this as an example for description. The larger the ratio, that is, the lower the ultrasonic signal regularity and the higher the variability, the higher the possibility of the existence of the corresponding tumor tissue. When the ultrasonic suppression amount of a certain ultrasonic block in a to-be-trained image is relatively high, the probability of the existence of a tumor at this time is relatively high. At this time, even if the tumor is occult (growing inwards) in the image and its edge characteristics are not obvious, based on this index, subsequent training analysis can be carried out, and the weight of the pixels in this area during the training process can be increased, with the aim of obtaining more intelligent network training parameters.

[0072] Preset the suppression amount threshold to 0.8. Take this as an example for description.

[0073] Denote the ultrasonic block corresponding to the ultrasonic echo signal with a suppression amount greater than or equal to the preset suppression amount threshold as a high-suppression-amount ultrasonic block.

[0074] Module 103: Tumor tissue heterogeneity analysis module.

[0075] This module is used to determine the tumor visibility of each high-suppression-amount ultrasonic block according to the difference in the gray values of the pixel points between each high-suppression-amount ultrasonic block and its adjacent ultrasonic blocks; and determine the tumor tissue heterogeneity of each high-suppression-amount ultrasonic block according to the tumor visibility and the suppression amount of each high-suppression-amount ultrasonic block.

[0076] It should be noted that: if there are high-suppression ultrasound blocks in the ultrasound images to be trained, it is necessary to analyze the pixel edges in the images to obtain the tissue heterogeneity of the pixel points within the blocks, and then determine the upsampling weight values of the pixel points in the suspected tumor regions during training. That is, obtain the training set that meets the ultrasound suppression amount and its corresponding high-suppression ultrasound blocks, and analyze the tumor tissue heterogeneity within the blocks. This heterogeneity means that even during the doctor's operation, some tumors are visually difficult to identify or detect, but after pre-analysis in the ultrasound dimension, for the ultrasound blocks in the suspected tumor regions, it is necessary to focus on analyzing their image characteristics. For example, for occult (inward-growing) tumors, compared with the tissue regions in the image, due to the effect of internal tumor filling, they are generally darker visually. And kidney tumors often exhibit different densities from the surrounding normal kidney tissues. A common manifestation is that the density of the tumor region is higher or lower, depending on whether the tumor contains components such as calcification, hemorrhage, and fat. For example, calcified tumors usually exhibit a higher density and appear as high-density regions. Therefore, such high-density and darker characteristics, which make the details of the image difficult for doctors to capture, can be further verified through robot vision, that is, tumor visibility.

[0077] Preferably, in an embodiment of the present invention, the method for obtaining the tumor tissue heterogeneity of each high-suppression ultrasound block includes:

[0078] Calculate the mean value of the gray values of all pixel points within each ultrasound block as the gray mean value of each ultrasound block.

[0079] Calculate the mean value of the gray values of all pixel points in the ultrasound image where any high-suppression ultrasound block is located as the gray mean value of this ultrasound image. Denote the ratio of the gray mean value of this ultrasound image to the gray mean value of this any high-suppression ultrasound block as the first ratio. In this ultrasound image, calculate the absolute value of the difference between the gray mean value of this any high-suppression ultrasound block and the gray mean value of each of its adjacent ultrasound blocks. Denote the sum value of the absolute values of the differences between the gray mean value of this any high-suppression ultrasound block and the gray mean values of all its adjacent ultrasound blocks as the first sum value. Denote the normalized value of the ratio of the first ratio to the first sum value as the tumor visibility of this any high-suppression ultrasound block.

[0080] It should be noted that: in this embodiment, use a linear normalization function to normalize the ratio of the first ratio to the first sum value to Within the interval, if the first sum value is 0, then set the first sum value to 1 to ensure the ratio holds, and the following will be described by taking this as an example. The smaller the ratio of the gray mean value of the high-inhibition ultrasound block to the gray mean value of the ultrasound image, the darker the high-inhibition ultrasound block, and the more likely a tumor exists. Therefore, the larger the first ratio, the higher the tumor visibility of the corresponding high-inhibition ultrasound block. Secondly, by quantitatively analyzing the degree of gray closeness of the surrounding ultrasound blocks, that is, the smaller the first sum value, the closer the gray properties of the surrounding ultrasound blocks, and then the higher the local pixel density (in the high-density area where pixels are concentrated, which is manifested as the closeness of gray levels in details).

[0081] Preset the first allocation weight as 0.6 and the second allocation weight as 0.4, and the following will be described by taking this as an example, where the sum value of the preset first allocation weight and the preset second allocation weight is 1.

[0082] Calculate the product of the inhibition amount of any high-inhibition ultrasound block and the preset first allocation weight, denoted as the first product. Calculate the product of the tumor visibility of the same high-inhibition ultrasound block and the preset second allocation weight, denoted as the second product. Take the sum value of the first product and the second product as the tumor tissue heterogeneity of any high-inhibition ultrasound block.

[0083] Module 104: Renal tumor region recognition module.

[0084] This module is used to determine the upsampling weight of each pixel point in the ultrasound image according to the size of the tumor tissue heterogeneity; train the training set according to the upsampling weight to obtain a trained semantic segmentation network model; use the trained semantic segmentation network model to segment the renal tumor region in the ultrasound image.

[0085] It should be noted that: each image to be trained contains several ultrasound blocks, some of which have high inhibition, and the higher the corresponding tissue heterogeneity in the block, the higher the possibility of a tumor, that is, the more weight allocation is required during the training process. In the semantic segmentation network, during the training stage, when upsampling and restoring the edges of the image, each pixel point has the same weight value. To improve the discrimination accuracy of the network for occult (inward-growing) tumors, the weight of the pixel points in the ultrasound blocks with increased tissue heterogeneity during the upsampling process in the network training is selected to optimize the training results.

[0086] Preferably, in an embodiment of the present invention, the method for obtaining the renal tumor region includes:

[0087] Preset the basic weight as 1, and the following will be described by taking this as an example.

[0088] Set the upsampling weight of each pixel point in any high suppression amount ultrasound block to the sum value of the preset basic weight and the tumor tissue heterogeneity of the any high suppression amount ultrasound block.

[0089] Set the upsampling weight of each pixel point not in the high suppression amount ultrasound block to the preset basic weight.

[0090] It should be noted that for the pixel points in the high suppression amount ultrasound block, on the basis of the basic weight, the upsampling rate is increased, and the replication ratio of this part of the ultrasound block is increased during the upsampling process (the upsampling algorithm, which is a well-known technology), that is, the sampling result highlights this ultrasound block more.

[0091] Thus, the upsampling weight of each pixel point in each frame of ultrasound image is obtained.

[0092] According to the upsampling weight of each pixel point in each frame of ultrasound image in the training set, train the training set to obtain a trained semantic segmentation network model. Use the trained semantic segmentation network model to segment the renal tumor area in the ultrasound image.

[0093] It should be noted that: the schematic diagram of the overview of the optimization training process for renal tumor recognition is as Figure 4 shown Figure 4 In the figure, convolutional pooling is performed on the training set (including multiple frames of ultrasound images (Photo)), and then the system analyzes the tissue heterogeneity ultrasound blocks in the ultrasound blocks, that is, the high suppression amount ultrasound blocks, and finally performs upsampling optimization. Subsequently, the trained semantic segmentation network model is configured into the robot recognition system. During the process of tumor recognition, this model has a relatively sensitive response ability to occult (inward-growing) renal tumors. In the actual detection and recognition process of robot surgery, when the ultrasound suppression amount of an ultrasound block exceeds the threshold specified during the training process, semantic segmentation processing is performed on it to obtain the machine-assisted segmentation result of the tumor edge. In the imaging system, the recognition result border of semantic segmentation is generated and marked as the machine-assisted renal tumor recognition result, serving the purpose of impact analysis during the renal tumor resection process.

[0094] It should be further noted that: the segmentation neural network used in this embodiment is the Mask R-CNN neural network, and the dataset used is the ultrasonic image dataset. Among them, Mask R-CNN is a well-known technology, and the specific method will not be introduced here. The Chinese full name of Mask R-CNN is "Mask Region-based Convolutional Neural Network", and the English full name is "Mask Region-based Convolutional Neural Network". The pixel points to be segmented are divided into 2 categories, that is, the label annotation process corresponding to the training set is: for the single-channel semantic label, the pixel points at the corresponding positions are labeled 0 if they belong to the normal region, and labeled 1 if they belong to the renal tumor region. The task of the network is classification, so the loss function used is the cross-entropy loss function, which is a well-known technology.

[0095] So far, the present invention is completed.

[0096] In summary, in the embodiment of the present invention, ultrasonic images during renal tumor resection are obtained, a training set is constituted by part of the ultrasonic images, the ultrasonic images are equally divided into several ultrasonic blocks, where each ultrasonic block corresponds to an ultrasonic echo signal, the energy of the ultrasonic echo signal at different frequency components is obtained, so as to determine the suppression amount of the ultrasonic echo signal, and the ultrasonic blocks with high suppression amount are screened out. Then, the tumor tissue heterogeneity of the ultrasonic blocks with high suppression amount is determined to obtain the upsampling weight of the pixel points in the ultrasonic image, and thus the training set is trained to obtain a trained semantic segmentation network model for segmenting the renal tumor region in the ultrasonic image. The present invention improves the accuracy of renal tumor region recognition through the adaptive upsampling weight, thereby ensuring the effect of renal tumor resection.

[0097] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intraoperative robotic imaging analysis system for renal tumor resection, characterized in that, The system includes the following modules: Renal tumor resection image acquisition module: used to obtain several frames of ultrasonic images during renal tumor resection, and form a training set with partial ultrasonic images; equally divide each frame of ultrasonic image into several ultrasonic blocks; where each ultrasonic block corresponds to an ultrasonic echo signal; High suppression amount ultrasonic block recognition module: used to obtain the energy of each ultrasonic echo signal at different frequency components; determine the suppression amount of each ultrasonic echo signal according to the magnitude of the energy at different frequency components and the extreme value difference in the ultrasonic echo signal; screen out high suppression amount ultrasonic blocks according to the magnitude of the suppression amount; Tumor tissue heterogeneity analysis module: used to determine the tumor visibility of each high suppression amount ultrasonic block according to the pixel gray value difference between each high suppression amount ultrasonic block and adjacent ultrasonic blocks; determine the tumor tissue heterogeneity of each high suppression amount ultrasonic block according to the tumor visibility and the magnitude of the suppression amount of each high suppression amount ultrasonic block; Renal tumor region recognition module: used to determine the upsampling weight of each pixel point in the ultrasonic image according to the magnitude of the tumor tissue heterogeneity; train the training set according to the upsampling weight to obtain a trained semantic segmentation network model; use the trained semantic segmentation network model to segment the renal tumor region in the ultrasonic image; The determination of the suppression amount of each ultrasonic echo signal includes: Equally divide each ultrasonic echo signal into several ultrasonic echo signal segments; Calculate the mean value of all maximum values in each ultrasonic echo signal segment as the amplitude of each ultrasonic echo signal segment; Determine the ultrasonic variability of each ultrasonic echo signal according to the difference between the amplitudes of the ultrasonic echo signal segments in each ultrasonic echo signal; Determine the ultrasonic regularity of each ultrasonic echo signal according to the magnitude of the energy of each ultrasonic echo signal at different frequency components; Determine the suppression amount of each ultrasonic echo signal according to the ultrasonic variability and ultrasonic regularity of each ultrasonic echo signal; The determination of the ultrasonic regularity of each ultrasonic echo signal includes: For any ultrasonic echo signal, obtain the maximum value among the energies at all different frequency components, and then obtain the sum value of the energies at all different frequency components. Denote the ratio of the maximum value to the sum value as the ultrasonic regularity of the any ultrasonic echo signal.

2. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, wherein The determination of the ultrasonic variability of each ultrasonic echo signal includes: In the amplitudes of all ultrasonic echo signal segments of each ultrasonic echo signal, take the difference between the maximum amplitude and the minimum amplitude as the ultrasonic variability of each ultrasonic echo signal.

3. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, characterized in that, The determination of the suppression amount of each ultrasonic echo signal according to the ultrasonic variability and ultrasonic regularity of each ultrasonic echo signal includes: Take the normalized value of the ratio of the ultrasonic variability and ultrasonic regularity of each ultrasonic echo signal as the suppression amount of each ultrasonic echo signal.

4. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, wherein The screening out of high suppression amount ultrasonic blocks includes: Denote the ultrasonic block corresponding to the ultrasonic echo signal with a suppression amount greater than or equal to the preset suppression amount threshold as a high suppression amount ultrasonic block.

5. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, wherein, The determination of the tumor visibility of each high suppression amount ultrasonic block includes: Calculate the mean value of the gray values of all pixel points in each ultrasonic block as the gray mean value of each ultrasonic block; Calculate the mean of the gray values of all pixel points in each frame of the ultrasound image as the gray mean of each frame of the ultrasound image; Determine the tumor visibility of each high-suppression ultrasound block according to the gray mean of each frame of the ultrasound image and each ultrasound block.

6. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 5, wherein, The determining the tumor visibility of each high-suppression ultrasound block according to the gray mean of each frame of the ultrasound image and each ultrasound block includes: For any high-suppression ultrasound block, record the ratio of the gray mean of the ultrasound image where the high-suppression ultrasound block is located to the gray mean of the high-suppression ultrasound block as the first ratio, record the sum of the absolute values of the differences between the gray mean of the high-suppression ultrasound block and the gray means of all adjacent ultrasound blocks as the first sum value, and record the normalized value of the ratio of the first ratio to the first sum value as the tumor visibility of any high-suppression ultrasound block.

7. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, wherein The determining the tumor tissue heterogeneity of each high-suppression ultrasound block includes: For any high-suppression ultrasound block, record the product of the suppression amount of the high-suppression ultrasound block and the preset first allocation weight as the first product, record the product of the tumor visibility of the high-suppression ultrasound block and the preset second allocation weight as the second product, and use the sum value of the first product and the second product as the tumor tissue heterogeneity of any high-suppression ultrasound block; where the sum of the preset first allocation weight and the preset second allocation weight is 1.

8. The intraoperative robotic imaging analysis system for renal tumor resection according to claim 1, wherein The determining the upsampling weight of each pixel point in the ultrasound image includes: Set the upsampling weight of each pixel point in any high-suppression ultrasound block to be the sum of the preset basic weight and the tumor tissue heterogeneity of any high-suppression ultrasound block; Set the upsampling weight of each pixel point not in the high-suppression ultrasound block to the preset basic weight.

Images