Tumor boundary quantitative division method and device and electronic equipment

Abstract

The invention provides a tumor boundary quantitative division method and device and electronic equipment, and relates to the field of image processing, and the method comprises the steps: detecting whether there is a difference between two frames of images of fixed interval frames in an ICG fluorescence image; in response to the difference greater than a specified difference degree, performing unsupervised clustering on pixel points of x in the ICG fluorescence image according to gray values by using an unsupervised clustering method to obtain a plurality of categories; setting a region weight for each category image region according to the variance of pixel values in the category image region corresponding to each category and the number of pixels, and determining a region parameter corresponding to each category image region based on the region weight, the mean value of the pixel values and the variance; and quantifying the tumor boundary in the ICG fluorescence image based on the region weight and the region parameter to obtain a fluorescence image after tumor boundary quantification, fusing the fluorescence image after tumor boundary quantification with the visible light image, and obtaining and displaying a final tumor boundary quantitative division result image in a graphical user interface.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a tumor boundary quantitative division method and device and electronic equipment. BACKGROUND

[0002] At present, surgical resection is one of the most effective tumor treatment options. In this process, the doctor distinguishes the tumor and normal tissue according to visual observation, and then resects the tumor tissue along the tumor boundary, both completely resecting the tumor tissue and preserving the normal tissue as much as possible to achieve the best treatment effect. The accurate determination of the tumor and its boundary requires the doctor to have a lot of clinical experience, which increases the technical threshold of tumor resection surgery. In order to reduce the technical threshold and improve the accuracy of tumor resection, the fluorescence surgery navigation technology using indocyanine green (ICG) as a tracer is widely used. Using camera imaging technology, ICG can mark tumor tissue in real time in the form of a fluorescence image during surgery to help the doctor quickly locate the tumor. However, the ICG fluorescence image is blurred in marking the tumor boundary, and the doctor still needs to rely on accumulated clinical experience to resect the tumor boundary. In order to accurately mark the tumor boundary in the ICG fluorescence image, the tumor boundary quantification technology uses the doctor to mark and DNR (Difference to Noise Ratio) to quantify the tumor boundary.

[0003] Although this method can accurately mark the tumor boundary on the ICG fluorescence image, the doctor needs to mark the area where the non-tumor tissue but still has ICG fluorescence on the ICG fluorescence image multiple times during use. This auxiliary marking will interrupt the doctor's surgery process and bring inconvenience to the doctor's use of tumor boundary quantification. Therefore, the current tumor boundary quantification method has low operation convenience. SUMMARY

[0004] The purpose of the present application is to provide a tumor boundary quantitative division method, device and electronic equipment to solve the technical problem of low operation convenience of the current tumor boundary quantification method.

[0005] In a first aspect, the present application provides a tumor boundary quantitative division method, which comprises: obtaining a plurality of intraoperative tumor tissue images; the intraoperative tumor tissue images include ICG fluorescence images and visible light images; detecting whether there is a difference between two images of fixed interval frames in the ICG fluorescence image through a twin network; in response to the difference being greater than a specified difference degree, unsupervised clustering is performed on the pixels in the ICG fluorescence image according to the gray value, to obtain a plurality of categories, so as to simulate the process of manually marking the region of the ICG fluorescence image by unsupervised clustering; According to the variance and the number of pixels of the pixel value in each category image region, a region weight is set for each category image region, and based on the region weight, the mean value and the variance of the pixel value, a region parameter corresponding to each category image region is determined. Based on the region weight and the region parameter, the boundary of the tumor in the ICG fluorescence image is quantified to obtain a tumor boundary quantized fluorescence image, and the tumor boundary quantized fluorescence image is fused with the visible light image to obtain a final tumor boundary quantitative division result image which is displayed in a graphical user interface.

[0006] In one possible implementation, after determining the region parameter corresponding to each category image region based on the region weight, the mean value and the variance of the pixel value, the method further comprises: Adaptively quantifying the boundary of the tumor in the image based on the region parameter; In response to detecting an interactive operation of marking part of the image region in x in the ICG fluorescence image, the region weight is updated by an incremental learning strategy according to the interactive operation to obtain an updated region weight.

[0007] In one possible implementation, the plurality of intraoperative tissue images with tumors is a dual-channel video stream obtained from a dual-channel camera, wherein the dual-channel video stream is a visible light video stream and an ICG fluorescence video stream respectively; the twin network comprises two sub-networks with shared weights and the same structure, and a fusion discrimination module; The twin network detects whether there is a difference between two images of the fixed interval frames in the ICG fluorescence image, comprising: Reading each frame of the ICG fluorescence image in the ICG fluorescence video stream; Scaling and normalizing the ICG fluorescence image to obtain a processed image, and storing the processed image in a cache queue with a specified length; In response to the storage length of the cache queue exceeding the specified length, the last frame of image and the first frame of image in the queue are taken out from the cache queue, and the last frame of image and the first frame of image in the queue are input to the trained twin network in the form of image pairs, so as to extract the first image feature of the last frame of image and the second image feature of the first frame of image by the two sub-networks with shared weights. obtaining, by the fusion discrimination module, a similarity probability between the last frame image and the first frame image of the queue based on the first image feature and the second image feature; wherein the fusion discrimination module fuses the first image feature and the second image feature by means of bitwise multiplication and outputs the fused feature through a linear transformation layer and a Sigmoid activation function; determining whether there is a difference between the last frame image and the first frame image of the queue according to the similarity probability and a specified threshold value; the specified threshold value corresponds to the specified difference degree; if the similarity probability is greater than or equal to the specified threshold value, it is determined that the difference between the two frames of images is greater than the specified difference degree; if the similarity probability is less than the specified threshold value, it is determined that the difference between the two frames of images is less than the specified difference degree, and the step of detecting whether there is a difference between the two frames of images of the fixed interval frame in the ICG fluorescence image by the twin network is repeatedly executed.

[0008] In one possible implementation, the unsupervised clustering method includes a K-Means algorithm; the unsupervised clustering of the pixel points of x in the ICG fluorescence image according to the gray value by the unsupervised clustering method obtains a plurality of categories, including: K pixel points in x of the ICG fluorescence image are randomly selected as initial cluster centers; the gray difference values of each pixel in x with the K cluster centers are calculated by the following formula: wherein, is the gray value of any pixel point in x, is the gray value of any cluster center; any pixel point in x and the corresponding cluster center are represented by the K gray difference values, and the pixel point is assigned to the target category with the smallest gray difference value, so that all pixel points on x in the ICG fluorescence image correspond to a category label indicating the category to which the pixel point belongs, and a plurality of clustered categories are obtained.

[0009] In one possible implementation, the region weight of each category image region is set according to the variance and the number of pixel values of each category image region, and the region parameter corresponding to each category image region is determined based on the region weight, the mean value of the pixel value and the variance, including: the variance and the mean value of the pixel value in each category image region corresponding to each category are calculated, and the mean value and the variance of the plurality of categories corresponding to the number of categories are obtained; sort the individual array values in descending order of the mean values to obtain a sorting result; the sorting result is represented by a first expression as follows: , ; wherein, is the mean value and variance of the i-th pixel category; set a region weight for each of the category image regions according to the variance and the number of pixels in the category image region; wherein, the variance represents a uniform distribution, the number of pixels represents the area of the category image region, the larger the area and the more uniform the distribution, the higher the region weight; the region weight is calculated as follows: ; wherein, represents the region weight, is the number of pixels in the category image region, is a constant, set to 1e-4; convert the first expression into a second expression as follows based on the region weight: , to obtain a region parameter corresponding to each of the category image regions.

[0010] In one possible implementation, the quantification of the boundary of the tumor in the ICG fluorescence image based on the region weight and the region parameter to obtain a tumor boundary quantified fluorescence image includes: calculate a total weight and an accumulated weight based on the second expression; wherein, the total weight is , is the number of categories; the accumulated weight is ; take 0.2, 0.4, 0.6, 0.8 of the total weight as key thresholds, find four target positions in the sequence corresponding to the second expression according to the accumulated weight, the first time the key thresholds are reached or exceeded, establish a mapping relationship and adjust the pixel label to [0, 1, 2, 3, 4], reserve the high weight corresponding image bright area higher than the specified weight and merge the low weight corresponding image dark area lower than the specified weight according to the four target positions, to obtain a tumor boundary quantified fluorescence image.

[0011] In one possible implementation, the tumor boundary quantified fluorescence image is fused with the visible light image to obtain and display a final tumor boundary quantitative division result image in a graphical user interface, including: ​The fluorescence image after quantizing the tumor boundary is mapped to the pixel values ​​of the fluorescence image using a preset color level conversion strategy to obtain a color level converted fluorescence image. The color level converted fluorescence image is then superimposed with the RGB components of the visible light image to obtain a fused image. The fused image is used as the final tumor boundary quantitative division result image and is displayed in real time in the graphical user interface provided by the user terminal.

[0012] Secondly, this application provides a tumor boundary quantitative delineation device, the device comprising: The acquisition unit is used to acquire multiple intraoperative tissue images with tumors; the intraoperative tissue images include ICG fluorescence images and visible light images; The detection unit is used to detect, via a twin network, whether there is a difference between two frames of the ICG fluorescence image at a fixed interval; A clustering unit is used to perform unsupervised clustering on the pixels of x in the ICG fluorescence image according to gray values ​​in response to the difference being greater than a specified difference level, so as to obtain multiple categories, thereby simulating the process of manually marking regions in the ICG fluorescence image through unsupervised clustering. The determining unit is configured to set a region weight for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and to determine the region parameters corresponding to each category image region based on the region weight, the mean of the pixel values ​​and the variance; The quantization unit is used to quantize the boundary of the tumor in the ICG fluorescence image based on the region weight and the region parameter to obtain a fluorescence image after tumor boundary quantization, and to fuse the fluorescence image after tumor boundary quantization with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface.

[0013] Thirdly, this application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described in the first aspect above.

[0014] Fourthly, this application also provides a computer-readable storage medium storing computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method described in the first aspect above.

[0015] This application brings the following beneficial effects: This application provides a method, apparatus, and electronic device for quantitatively delineating tumor boundaries, capable of acquiring multiple intraoperative tissue images containing tumors; the intraoperative tissue images include ICG fluorescence images and visible light images; a twin network is used to detect whether there is a difference between two frames of the ICG fluorescence images at a fixed interval; in response to the difference being greater than a specified difference level, unsupervised clustering is used to unsupervisedly cluster the pixels of x in the ICG fluorescence images according to their gray values ​​to obtain multiple categories, thereby simulating the process of manually marking regions in the ICG fluorescence images through unsupervised clustering; a region weight is set for each category image region according to the variance of the pixel values ​​and the number of pixels in the category image region corresponding to each category, and a region parameter corresponding to each category image region is determined based on the region weight, the mean of the pixel values, and the variance; the boundary of the tumor in the ICG fluorescence image is quantified based on the region weight and the region parameter to obtain a quantized fluorescence image of the tumor boundary, and the tumor boundary is then... After tumor boundary quantification, the fluorescence image is fused with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface. In this scheme, unsupervised clustering of fluorescence images with large differences is used to simulate the effect of doctors manually marking regions on fluorescence images. Each clustered region is assigned a region weight, and the region classification is refined based on the calculation results of region weights and parameters. Finally, the fluorescence image and the visible light image are fused to obtain the final tumor boundary quantitative division result image. This adaptive tumor boundary quantification method marks the tumor boundary. Compared with existing tumor boundary quantification methods, it ensures the accuracy of tumor boundary quantification while reducing or even eliminating the interactive operation of doctors marking regions on fluorescence images during surgery by simulating doctors manually marking regions on fluorescence images. This makes it convenient for users to use the boundary quantification method and avoids or reduces surgical interruptions caused by the interactive operation of existing tumor boundary sharpening technology. It solves the technical problem of low operational convenience of current tumor boundary quantification methods.

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1A flowchart illustrating the method for quantitatively delineating tumor boundaries provided in this application embodiment; Figure 2 Another flowchart illustrating the tumor boundary quantitative delineation method provided in this application embodiment; Figure 3 The structure diagram of the twin network in the tumor boundary quantitative delineation method provided in the embodiments of this application; Figure 4 This is a schematic diagram of a tumor boundary quantitative delineation device provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] The terms "comprising" and "having," and any variations thereof, used in the embodiments of this application, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0021] Currently, methods for quantifying tumor boundaries are not very user-friendly. Therefore, this application provides a method, apparatus, and electronic device for quantitatively delineating tumor boundaries, which addresses the technical problem of the low ease of operation in current tumor boundary quantification methods.

[0022] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0023] Figure 1 This is a flowchart illustrating a method for quantitatively delineating tumor boundaries, as provided in an embodiment of this application. Figure 1 As shown, the method includes: Step S110: Acquire multiple intraoperative tissue images with tumors.

[0024] The intraoperative tissue images include ICG fluorescence images and visible light images.

[0025] As one possible implementation, the tumor boundary quantification method provided in this application includes six steps, such as... Figure 2 The examples shown are: image similarity detection; unsupervised image clustering; region weight and parameter calculation; incremental learning and region weight update; tumor boundary quantization and image fusion.

[0026] Step S120: Detect whether there is a difference between two frames of ICG fluorescence images with a fixed interval using a twin network.

[0027] For the specific process of image similarity detection, such as Figure 2 As shown, for example, the input consists of a dual-channel video stream acquired from a dual-channel camera, namely a white light video stream and an ICG fluorescence video stream. The purpose of image similarity detection is to check whether there are significant differences in the content of the ICG fluorescence video stream. When there are large changes in the content of the video stream, a series of subsequent processes are executed to simulate the operation of a doctor selecting a region of the ICG fluorescence image. In this embodiment, the image similarity detection uses a Siamese network to determine in real time whether there are significant changes between two images spaced ten frames apart. The model structure of this Siamese network is as follows: Figure 3 As shown, it includes two subnetworks (subnetwork 1 and subnetwork 2) with shared weights and identical structures, and a lightweight fusion and discrimination module. In this step, a cache queue is created to store images, and a Siamese network is used to determine whether there are significant differences between images at fixed intervals.

[0028] In one alternative implementation, a large number of image pairs are collected, and the Siamese network is trained to determine whether there are differences between the images. Fine-tuning can be performed on a pre-trained lightweight network to save training time.

[0029] As an example, the aforementioned intraoperative tissue images with tumors are dual-channel video streams acquired from a dual-channel camera, wherein the dual-channel video streams are a visible light video stream and an ICG fluorescence video stream, respectively; the Siamese network comprises two sub-networks with shared weights and identical structures, and a fusion discriminant module; the aforementioned detection of whether there is a difference between two frames of ICG fluorescence images at a fixed interval using the Siamese network may specifically include the following steps: Read each frame of ICG fluorescence image from the ICG fluorescence video stream; scale and normalize the ICG fluorescence image to obtain the processed image, and store the processed image in a buffer queue of a specified length; In response to the storage length of the cache queue exceeding the specified length, the last frame image and the first frame image of the queue are retrieved from the cache queue, and the last frame image and the first frame image of the queue are input into the trained Siamese network as an image pair, so as to extract the first image feature of the last frame image and the second image feature of the first frame image of the queue through the two sub-networks with shared weights. Based on the first image features and the second image features, the similarity probability between the last frame image and the first frame image in the queue is obtained through the fusion discrimination module; wherein, the fusion discrimination module fuses the first image features and the second image features by bitwise multiplication and outputs the fused features through a linear transformation layer and a Sigmoid activation function; The system determines whether there is a difference between the last frame and the first frame of the queue based on the similarity probability and a specified threshold. The specified threshold corresponds to a specified degree of difference. If the similarity probability is greater than or equal to the specified threshold, the difference between the two frames is determined to be greater than the specified degree of difference. If the similarity probability is less than the specified threshold, the difference between the two frames is determined to be less than the specified degree of difference. The system then repeats the step of detecting whether there is a difference between two frames of ICG fluorescence images at fixed intervals using a Siamese network.

[0030] For example, each frame of the video stream is read, scaled to 1 / 4 of the original image, and normalized. The processed image is then stored in a buffer queue of length 11. When the queue length exceeds 11, the last frame (frame t) and the first frame (frame t-10) are retrieved from the queue. Figure 3 As shown, the input is given as image pairs (Input 1, Input 2) to the trained Siamese network. Two sub-networks (sub-network 1, sub-network 2) sharing weights extract features from the two frames (Feature 1: Features 1, Feature 2: Features 2). These features are then processed by a fusion classifier to obtain the similarity probability between the two frames, with a probability value range of 0-1. A predefined threshold (0.7 in this embodiment) is then used to determine if there is a significant change between the two frames. If the probability value is greater than or equal to the threshold, a significant change is considered to exist, triggering step 2; otherwise, no significant change is considered, and step 1 is repeated. In this embodiment, Features 1 and Features 2 are fused using bitwise multiplication. The fused features are then output through a linear transformation layer and a sigmoid activation function.

[0031] Step S130: In response to the difference being greater than a specified difference level, unsupervised clustering is used to perform unsupervised clustering on the pixels of x in the ICG fluorescence image according to their gray values ​​to obtain multiple categories, so as to simulate the process of manually marking regions in the ICG fluorescence image through unsupervised clustering.

[0032] It should be noted that unsupervised clustering is a machine learning technique used to label data into different categories based on certain features without requiring prior human intervention. As an example, unsupervised clustering includes the K-Means algorithm; the above method uses unsupervised clustering to unsupervisedly cluster pixels x in an ICG fluorescence image according to their grayscale values, obtaining multiple categories. Specifically, this may include the following steps: K pixels are randomly selected from x in the ICG fluorescence image as initial cluster centers; the grayscale difference between each pixel in x and the K cluster centers is calculated using the following formula: ,in, It is the grayscale value of any pixel in x. It is the gray value of any cluster center; K gray-level difference values ​​are used to represent the gray-level difference between any pixel in x and its corresponding cluster center. The pixels are then assigned to the target category with the smallest gray-level difference value, so that all pixels in x in the ICG fluorescence image have a category label to indicate their category, resulting in multiple clustered categories.

[0033] In practical applications, such as Figure 2 As shown, pixels in the ICG fluorescence image x are clustered unsupervised according to their grayscale values. For example, the K-Means algorithm is used to cluster images x, first randomly selecting pixels from x... Using 100 pixels as the initial cluster centers, then calculate the relationship between each pixel in x and these 100 pixels. Gray-scale differences among cluster centers: ,in It is the grayscale value of any pixel in x. This is the gray value of any cluster center. Thus, any pixel in x will have... Each difference value represents the grayscale difference between the pixel and its corresponding cluster center, and the pixel is assigned to the category with the smallest difference value. In this way, all pixels in image x will have a category label indicating their category. For example, in the embodiments of this application... At this point, the system will detect whether there is user interaction, such as... Figure 2 As shown, if there is no interaction, the calculation process of region weights and parameters in step S140 is executed; if there is interaction, the incremental learning and region weight update process in step S145 is executed.

[0034] Step S140: Set region weights for each category image region based on the variance of pixel values ​​and the number of pixels in the corresponding category image region for each category, and determine the region parameters corresponding to each category image region based on the region weights, the mean and variance of pixel values.

[0035] In one possible implementation, the above-mentioned setting of region weights for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and determining the region parameters corresponding to each category image region based on the region weights, the mean and variance of pixel values, may specifically include the following steps: Calculate the variance and mean of pixel values ​​in the image region corresponding to each category, obtaining the mean and variance of arrays corresponding to the number of categories; sort the array values ​​in descending order of mean to obtain the sorted result; the sorted result is represented by the following first expression: , ;in, It is the first The mean and variance of each pixel category; A region weight is assigned to each category image region based on the variance and the number of pixels within the category image region. Here, variance represents the uniformity of distribution, and the number of pixels represents the area of ​​the category image region. A larger region area and a more uniform distribution result in a higher region weight. The region weight is calculated as follows: ; in, Indicates the regional weight. The number of pixels in the category image region. As a constant, set to 1e-4; based on the region weights, the first expression is transformed into the following second expression: This yields the region parameters corresponding to each category of image region.

[0036] like Figure 2 As shown, for the process of calculating region weights and parameters, for example, the image is... All pixels are divided into grayscale values. There are 10 categories. Then, the mean and variance of the pixel values ​​for each category are calculated, resulting in 10 sets of means and variances. These 10 sets of values ​​are then sorted from largest to smallest by mean, as shown in the expression: , .in It is the first The mean and variance of each pixel category. In this embodiment, each set of mean and variance will have a weight. A higher weight indicates a larger and more evenly distributed area. Thus, the above expression can be transformed into: Use these parameters as input values ​​for step S150 below. When an interaction is detected in the process, modify them according to step S145 below. arrive The value of .

[0037] Step S150: The tumor boundary in the ICG fluorescence image is quantified based on the region weight and region parameters to obtain a fluorescence image after tumor boundary quantification. The fluorescence image after tumor boundary quantification is then fused with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface.

[0038] In one optional implementation, the above-mentioned quantization of the tumor boundary in the ICG fluorescence image based on region weights and region parameters to obtain a fluorescence image with quantized tumor boundaries may specifically include the following steps: The total weight and cumulative weight are calculated based on the second expression; where the total weight is... , The number of categories; the cumulative weight is Using 0.2, 0.4, 0.6, and 0.8 of the total weight as key thresholds, the four target positions that first reach or exceed the key thresholds are found in the sequence corresponding to the second expression according to the cumulative weight. A mapping relationship is established and the pixel labels are adjusted to [0, 1, 2, 3, 4]. Based on the four target positions, the bright areas of the image corresponding to the high weights above the specified weights are retained, and the dark areas of the image corresponding to the low weights below the specified weights are merged to obtain the fluorescence image after tumor boundary quantification.

[0039] like Figure 2 As shown, the following parameters are obtained through step S140 above for quantifying the tumor boundary: Calculate the total weight and cumulative weight Four key thresholds are set. In this embodiment, the key thresholds are 0.2, 0.4, 0.6 and 0.8 of the total weight. The position that first reaches or exceeds the key threshold in the sequence is found according to the cumulative weight. A mapping is established and the pixel label is adjusted to [0,1,2,3,4]. Based on the four positions, the bright areas with high weight are retained and the dark areas with low weight are merged.

[0040] As an optional implementation, the above-mentioned fusion of the fluorescence image after tumor boundary quantification with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface may specifically include the following steps: By using a color level conversion strategy and a preset color level conversion relationship, the fluorescence image after quantizing the tumor boundary is mapped to the pixel values ​​of the fluorescence image to obtain a color level converted fluorescence image. The color level converted fluorescence image is then superimposed with the corresponding RGB components of the visible light image to obtain a fused image. The fused image is used as the final tumor boundary quantitative division result image and is displayed in real time in the graphical user interface provided by the user terminal.

[0041] like Figure 2As shown, for image fusion, this embodiment of the application utilizes a color level conversion strategy to superimpose the RGB components of the color-level converted fluorescence image and the visible light image to form a fused image. Finally, the fused image is output to the user terminal interface in real time. For example, a preset color level conversion table is used to complete the pixel value mapping on the fluorescence image, which is then superimposed with the visible light image and output to the user terminal.

[0042] In existing technologies, tumor boundary sharpening techniques require doctors to select non-tumor areas with fluorescence on ICG fluorescence images as input for the algorithm to calculate tumor boundaries. However, when the calculation results need to be corrected for errors due to significant image changes, doctors need to interrupt the surgery to make corrections.

[0043] To reduce the occurrence of the above situations, this application proposes an adaptive intraoperative tumor boundary quantification method based on unsupervised clustering and incremental learning. This method simulates the effect of a surgeon manually marking regions on fluorescence images by performing unsupervised clustering on fluorescence images with significant differences. Each clustered region is assigned a weight, and the region classification is refined based on the calculated weights and parameters. Finally, the fluorescence image and the visible light image are fused to obtain the final tumor boundary quantification result image. This adaptive tumor boundary quantification method, compared to existing methods, ensures the accuracy of tumor boundary quantification while reducing or even eliminating the interactive operations required by surgeons during tumor boundary quantification by simulating manual marking on fluorescence images. This makes the boundary quantification method more convenient for users, avoiding or reducing surgical interruptions caused by the interactive operations of existing tumor boundary sharpening techniques, and solving the technical problem of low operational convenience in current tumor boundary quantification methods.

[0044] In some embodiments, after determining the region parameters corresponding to each category of image region based on the region weights, the mean and variance of pixel values, the method may further include the following steps: adaptively quantizing the boundary of the tumor in the image based on the region parameters; in response to detecting an interactive operation that marks a portion of the image region x in the ICG fluorescence image, correcting and updating the region weights according to the interactive operation through an incremental learning strategy to obtain the corrected and updated region weights.

[0045] The aforementioned incremental learning refers to the process in machine learning of absorbing new knowledge while retaining, integrating, and optimizing old knowledge to avoid catastrophic forgetting of old knowledge. In practical applications, after determining the region parameters corresponding to each category of image region based on region weights, the mean and variance of pixel values, the calculated parameters are adaptively quantized to define the tumor boundary, and the region weights are adaptively adjusted based on user interaction for incremental learning. Specifically, as shown... Figure 2As shown, if the doctor determines that there is a large error, the doctor will then perform an interactive operation by selecting the area, and then execute step S145, which updates the weights based on the interactive operation through incremental learning; if the doctor determines that there is no large error and does not perform the interactive operation by selecting the area, then step S145, which is the step of updating the weights, will not be executed.

[0046] Therefore, when the default parameter values ​​of unsupervised clustering cannot meet the requirements during surgery, resulting in incorrect quantification of tumor boundaries, the incremental learning strategy of this application embodiment can use a small amount of doctor interaction to correct the default parameter values ​​in this application embodiment to adapt to the tumor boundary quantification task in this scenario.

[0047] For the specific process of incremental learning and region weight update, as exemplified in this application embodiment, the interaction involves selecting a portion of the image x. When user interaction occurs, the weights are updated using an alpha blending method. arrive The value of .

[0048] In this embodiment of the application, after assigning weights to each type of region, if there is a large error at this time, the doctor will intervene to select regions again. Incremental learning will be carried out through weight updates via user interaction to improve the accuracy of the clustering algorithm.

[0049] Figure 4 A schematic diagram of a device for quantitatively delineating tumor boundaries is provided. Figure 4 As shown, the tumor boundary quantitative delineation device 400 includes: The acquisition unit 401 is used to acquire multiple intraoperative tissue images with tumors; the intraoperative tissue images include ICG fluorescence images and visible light images; Detection unit 402 is used to detect whether there is a difference between two frames of the ICG fluorescence image with a fixed interval frame through a twin network; Clustering unit 403 is used to perform unsupervised clustering on the pixels of the ICG fluorescence image according to gray value in response to the difference being greater than a specified difference level, so as to obtain multiple categories, so as to simulate the process of manually marking regions in the ICG fluorescence image through unsupervised clustering. The determining unit 404 is configured to set a region weight for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and to determine the region parameters corresponding to each category image region based on the region weight, the mean of the pixel values ​​and the variance; The quantization unit 405 is used to quantize the boundary of the tumor in the ICG fluorescence image based on the region weight and the region parameter to obtain a fluorescence image after tumor boundary quantization, and to fuse the fluorescence image after tumor boundary quantization with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface.

[0050] The tumor boundary quantitative division device provided in this application embodiment has the same technical features as the tumor boundary quantitative division method provided in the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0051] An electronic device provided in this application embodiment, such as Figure 5 As shown, the electronic device 500 includes a processor 502 and a memory 501. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the method provided in the above embodiments.

[0052] See Figure 5 The electronic device also includes a bus 503 and a communication interface 504. The processor 502, the communication interface 504 and the memory 501 are connected through the bus 503. The processor 502 is used to execute executable modules, such as computer programs, stored in the memory 501.

[0053] The memory 501 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 504 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network.

[0054] Bus 503 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0055] The memory 501 is used to store programs. After receiving an execution instruction, the processor 502 executes the program. The method executed by the apparatus defined by the process disclosed in any of the preceding embodiments of this application can be applied to the processor 502 or implemented by the processor 502.

[0056] Processor 502 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 502 or by instructions in software form. The processor 502 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 501, and processor 502 reads the information from memory 501 and, in conjunction with its hardware, completes the steps of the above method.

[0057] Corresponding to the above-described method for quantitatively delineating tumor boundaries, this application also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are invoked and executed by a processor, they cause the processor to perform the steps of the above-described method for quantitatively delineating tumor boundaries.

[0058] The tumor boundary quantitative delineation device provided in this application embodiment can be specific hardware on a device or software or firmware installed on the device. The implementation principle and technical effects of the device provided in this application embodiment are the same as those in the foregoing method embodiments. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the foregoing method embodiments. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can all be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0059] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0060] For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0062] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the tumor boundary quantitative division method described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0065] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for quantitatively delineating tumor boundaries, characterized in that, The method includes: Multiple intraoperative tissue images with tumors were acquired; the intraoperative tissue images included ICG fluorescence images and visible light images. The difference between two frames of the ICG fluorescence image with a fixed interval is detected by using a twin network. In response to the difference being greater than a specified difference level, unsupervised clustering is used to perform unsupervised clustering on the pixels of x in the ICG fluorescence image according to their gray values ​​to obtain multiple categories, so as to simulate the process of manually marking regions in the ICG fluorescence image through unsupervised clustering. A region weight is set for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and a region parameter corresponding to each category image region is determined based on the region weight, the mean of the pixel values ​​and the variance; The tumor boundary in the ICG fluorescence image is quantified based on the region weight and the region parameter to obtain a fluorescence image after tumor boundary quantification. The fluorescence image after tumor boundary quantification is then fused with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface.

2. The method according to claim 1, characterized in that, After determining the region parameters corresponding to each category image region based on the region weights, the mean of the pixel values, and the variance, the method further includes: The boundary of the tumor in the image is adaptively quantized based on the region parameters; In response to the detection of an interactive operation that marks a portion of the image region x in the ICG fluorescence image, the region weights are corrected and updated using an incremental learning strategy based on the interactive operation to obtain the corrected and updated region weights.

3. The method according to claim 1, characterized in that, The multiple intraoperative tissue images with tumors are dual-channel video streams acquired from a dual-channel camera, wherein the dual-channel video streams are a visible light video stream and an ICG fluorescence video stream, respectively; the twin network includes two subnetworks with shared weights and identical structures and a fusion discrimination module; The step of detecting whether there is a difference between two frames of the ICG fluorescence image at a fixed interval using a twin network includes: Read each frame of the ICG fluorescence image from the ICG fluorescence video stream; The ICG fluorescence image is scaled and normalized to obtain a processed image, and the processed image is stored in a cache queue of a specified length. In response to the storage length of the cache queue exceeding the specified length, the last frame image and the first frame image of the queue are retrieved from the cache queue, and the last frame image and the first frame image of the queue are input into the trained Siamese network in the form of an image pair, so as to extract the first image feature of the last frame image and the second image feature of the first frame image of the queue through the two sub-networks with shared weights. Based on the first image features and the second image features, the similarity probability between the last frame image of the queue and the first frame image of the queue is obtained by the fusion discrimination module; wherein, the fusion discrimination module fuses the first image features and the second image features by bitwise multiplication and outputs the fused features through a linear transformation layer and a Sigmoid activation function; The similarity probability and a specified threshold are used to determine whether there is a difference between the last frame image in the queue and the first frame image in the queue; the specified threshold corresponds to the specified degree of difference. If the similarity probability is greater than or equal to the specified threshold, then the difference between the two frames is determined to be greater than the specified degree of difference. If the similarity probability is less than the specified threshold, then it is determined that the difference between the two frames is less than the specified difference level, and the step of detecting whether there is a difference between the two frames of the ICG fluorescence image at a fixed interval through the Siamese network is executed cyclically.

4. The method according to claim 1, characterized in that, The unsupervised clustering method includes the K-Means algorithm; the unsupervised clustering method is used to unsupervisedly cluster the pixels of x in the ICG fluorescence image according to their gray values ​​to obtain multiple categories, including: K pixels are randomly selected from x in the ICG fluorescence image as the initial cluster centers; The grayscale difference between each pixel in x and the K cluster centers is calculated using the following formula: ,in, It is the grayscale value of any pixel in x. It is the gray value of any of the cluster centers; The grayscale difference between any pixel in x and its corresponding cluster center is represented by K grayscale difference values. The pixel is then assigned to the target category with the smallest grayscale difference value, so that all pixels in x in the ICG fluorescence image have a category label to indicate their category, resulting in multiple clustered categories.

5. The method according to claim 4, characterized in that, The step of setting a region weight for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and determining the region parameters corresponding to each category image region based on the region weight, the mean of the pixel values, and the variance, includes: Calculate the variance and mean of the pixel values ​​in the image region corresponding to each category to obtain the mean and variance of the arrays corresponding to the number of categories; The array values ​​are sorted in descending order of their mean values ​​to obtain a sorted result; the sorted result is represented by the following first expression: , ;in, It is the first The mean and variance of each pixel category; A region weight is assigned to each category image region based on the variance and the number of pixels in the category image region; wherein, the variance represents the uniformity of distribution, the number of pixels represents the area of ​​the category image region, and the larger the area and the more uniform the distribution, the higher the region weight; the region weight is calculated as follows: ； in, Indicates the weight of the region. The number of pixels in the image region of the category. It is a constant, set to 1e-4; Based on the region weights, the first expression is transformed into the following second expression: The region parameters corresponding to each category of image region are obtained.

6. The method according to claim 5, characterized in that, The step of quantizing the tumor boundary in the ICG fluorescence image based on the region weight and the region parameters to obtain a fluorescence image with quantized tumor boundaries includes: The total weight and cumulative weight are calculated based on the second expression; wherein, the total weight is... , The number of categories; the cumulative weight is ; Using 0.2, 0.4, 0.6, and 0.8 of the total weight as key thresholds, four target positions that first reach or exceed the key thresholds are found in the sequence corresponding to the second expression according to the cumulative weight. A mapping relationship is established and the pixel labels are adjusted to [0, 1, 2, 3, 4]. Based on the four target positions, the bright areas of the image corresponding to high weights above the specified weights are retained, and the dark areas of the image corresponding to low weights below the specified weights are merged to obtain the fluorescence image after tumor boundary quantization.

7. The method according to claim 1, characterized in that, The process of fusing the fluorescence image after quantifying the tumor boundary with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface includes: The fluorescence image after quantizing the tumor boundary is mapped to the pixel values ​​of the fluorescence image using a preset color level conversion strategy to obtain a color level converted fluorescence image. The color level converted fluorescence image is then superimposed with the RGB components of the visible light image to obtain a fused image. The fused image is used as the final tumor boundary quantitative division result image and is displayed in real time in the graphical user interface provided by the user terminal.

8. A device for quantitatively delineating tumor boundaries, characterized in that, The device includes: The acquisition unit is used to acquire multiple intraoperative tissue images with tumors; the intraoperative tissue images include ICG fluorescence images and visible light images; The detection unit is used to detect, via a twin network, whether there is a difference between two frames of the ICG fluorescence image at a fixed interval; A clustering unit is used to perform unsupervised clustering on the pixels of x in the ICG fluorescence image according to gray values ​​in response to the difference being greater than a specified difference level, so as to obtain multiple categories, thereby simulating the process of manually marking regions in the ICG fluorescence image through unsupervised clustering. The determining unit is configured to set a region weight for each category image region based on the variance of pixel values ​​and the number of pixels in the category image region corresponding to each category, and to determine the region parameters corresponding to each category image region based on the region weight, the mean of the pixel values ​​and the variance; The quantization unit is used to quantize the boundary of the tumor in the ICG fluorescence image based on the region weight and the region parameter to obtain a fluorescence image after tumor boundary quantization, and to fuse the fluorescence image after tumor boundary quantization with the visible light image to obtain and display the final tumor boundary quantitative division result image in the graphical user interface.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method according to any one of claims 1 to 7.

Images