Method, system and device for risk assessment of lesion resection based on endoscopic ultrasound images
Feature extraction of ultrasound endoscopic images through object detection and semantic segmentation models to evaluate the risk of perforation during digestive tract tumor resection, solve the problem of lack of accurate evaluation standards in the prior art, and improve surgical safety and evaluation accuracy.
Patent Information
- Application Number
- CN202411161591.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-23
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-23
AI Technical Summary
The existing ultrasound endoscopy technology lacks accurate perforation risk assessment standards during the resection of digestive tract tumors, making it difficult to formulate safe surgical plans in clinical practice, increasing the risk of surgery.
The ultrasonic endoscopic images are processed using the object detection model and the semantic segmentation model. The feature parameters include physical length feature parameters. These parameters are used to evaluate the perforation probability during the resection process, and a systematic risk assessment method is provided.
Accurate prediction of perforation risks in gastrointestinal tumor surgery, improve surgical safety, reduce the probability of intraoperative complications, provide doctors with objective and quantitative risk assessment basis, and make up for the prediction of perforation risks in the existing technology. Deficiencies in the aspects.
Smart Images

Figure CN118674720B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of EUS image processing, and in particular to a method, system, device and storage medium for risk assessment of lesion resection based on endoscopic ultrasound images. Background Art
[0002] Currently, endoscopic ultrasound (EUS) has become an important tool for diagnosing gastrointestinal lesions, especially in assessing the origin level and invasion depth of gastrointestinal tumors. Existing technologies have been using artificial intelligence, especially convolutional neural networks (CNN), to assist doctors in identifying and grading lesion areas.
[0003] For example, the patent with publication number CN115240019A discloses a method for identifying the origin layer of digestive tract tumors based on convolutional neural networks. By setting classification indicators, the origin layer of digestive tract tumors can be identified; the patent with publication number CN117218127A discloses an ultrasound endoscope-assisted monitoring system that can automatically identify and segment lesion areas and generate lesion-marked ultrasound images to help doctors make more accurate diagnoses; the patent with publication number CN116823695A proposes a gastric submucosal protrusion classification system based on multimodal image data and deep learning methods. By constructing a variety of AI models, the classification and screening of gastric submucosal protrusions can be achieved.
[0004] Although these existing technologies can assist doctors in grading and locating lesions to a certain extent, in actual treatment, especially during the resection of lesions, there is still a key challenge: how to accurately assess the risk of perforation that may occur during the resection. Current technical means mainly focus on the diagnosis and grading of lesions, but lack objective evaluation criteria for the risk of resection perforation, which has become a problem that needs to be solved in clinical practice. Therefore, it is necessary to develop a risk assessment method for lesion resection based on endoscopic ultrasound images. Summary of the invention
[0005] In view of the above problems, the present invention provides a method, system and device for assessing the risk of lesion resection based on ultrasonic endoscopic images, aiming to provide an accurate assessment method for the risk of resection perforation, fill the medical gap, provide reliable perforation risk assessment for clinicians, help doctors formulate safer surgical plans, and reduce surgical risks.
[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:
[0007] In a first aspect, the present invention provides a method for assessing the risk of lesion resection based on endoscopic ultrasound images, comprising:
[0008] Using the target detection model to perform target detection on the ultrasonic endoscopy image to obtain a first detection result and a second detection result;
[0009] The first target image is obtained by cropping the ultrasound endoscopic image according to the first detection result, and the second target image is obtained by cropping the ultrasound endoscopic image according to the second detection result;
[0010] Using a semantic segmentation model to segment the landmark area in the second target image to obtain a third target image;
[0011] Detecting scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly;
[0012] Extracting characteristic parameters of the ultrasonic endoscopic image, wherein the characteristic parameters include physical length characteristic parameters;
[0013] The perforation probability is evaluated according to the characteristic parameters.
[0014] In a preferred embodiment, the first detection result is the coordinate information of the effective area of the ultrasonic endoscopic image, and the first target image is the image of the effective area in the ultrasonic endoscopic image; the second detection result is the coordinates of the tumor area under the ultrasonic endoscope; and the second target image is the image of the tumor and its surrounding area in the ultrasonic endoscopic image.
[0015] In a preferred embodiment, the first target image includes scale segments in the ultrasonic endoscopy image; the detecting the scale segments in the first target image and calculating the physical length corresponding to each pixel accordingly specifically includes: cropping the first target image to obtain a sub-image with scale segments, processing the sub-image using an image smoothing algorithm and an edge extraction image processing algorithm, and then detecting the straight line segments in the processed sub-image, determining the pixel position information of the straight line segments, calculating the pixel distance between adjacent straight line segments, and calculating the physical length corresponding to each pixel based on the actual physical distance of the scale line spacing.
[0016] In a preferred embodiment, detecting the scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly specifically includes:
[0017] Cropping the first target image to obtain a first sub-image and a second sub-image, wherein the first sub-image has scale marks on the horizontal axis, and the second sub-image has scale marks on the vertical axis;
[0018] The first sub-image and the second sub-image are respectively processed as follows to obtain a first processed sub-image and a second processed sub-image in a one-to-one correspondence: both images are converted into grayscale images, the grayscale images are smoothed, and then edge detection is performed to obtain binary images;
[0019] Detecting straight line segments in the edge images of the first processed sub-image and the second processed sub-image;
[0020] The pixel position information of each straight line segment is obtained by performing line segment clustering analysis on the straight line segment;
[0021] Calculate the pixel distance between adjacent straight line segments in the same class of line segment cluster analysis;
[0022] The physical length corresponding to each pixel is calculated using the known actual physical distance of the scale line spacing, thereby obtaining the first physical marker parameter.
[0023] In a preferred embodiment, the semantic segmentation model is used to segment the landmark areas in the second target image, specifically: segmenting the landmark areas in the second target image and distinguishing and marking them, wherein the landmark areas include background, submucosal tumor area, mucosal layer area, mucosal muscular layer area, submucosal layer area, muscular layer propria area, and serous layer area.
[0024] In a preferred embodiment, the characteristic parameters include part or all of the first characteristic parameter, the second characteristic parameter, the third characteristic parameter, the fourth characteristic parameter, the fifth characteristic parameter, the sixth characteristic parameter, the seventh characteristic parameter, the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter;
[0025] The first characteristic parameter is the physical distance between the mucosal layer area and the submucosal tumor area, the second characteristic parameter is the physical distance between the submucosal tumor area and the serosa layer area, the second characteristic parameter includes the physical distance of the serosa layer area, and the third characteristic parameter is the ratio of the first characteristic parameter to the second characteristic parameter; the fourth characteristic parameter is the physical length of the submucosal tumor area, and the fifth characteristic parameter is the physical length of the short axis of the submucosal tumor area; the sixth characteristic parameter is the perforation confidence, and the seventh characteristic parameter is the non-perforation confidence; the eighth characteristic parameter is the site of tumor occurrence in the patient, the ninth characteristic parameter is the patient's age, and the tenth characteristic parameter is the patient's gender.
[0026] In a preferred embodiment, the extraction process of the first characteristic parameter, the second characteristic parameter, and the third characteristic parameter is as follows:
[0027] Calculating the main axis directions of the mucosal layer region and the submucosal tumor region, and rotating the third target image according to the main axis directions to obtain a rotated third target image; determining the distance between the mucosal layer region and the submucosal tumor region along the vertical direction, determining the distance between the submucosal tumor region and the bottom end of the serous layer region along the vertical direction, calculating the first characteristic parameter and the second characteristic parameter according to the first physical marker parameter, and calculating the third characteristic parameter;
[0028] The extraction process of the fourth characteristic parameter and the fifth characteristic parameter is as follows:
[0029] The minimum circumscribed rectangle algorithm is used to determine the long axis and short axis of the submucosal tumor region of the rotated third target image, wherein the short axis is perpendicular to the long axis, and the physical length of the long axis is calculated as the fourth characteristic parameter according to the first physical marker parameter, and the physical length of the short axis is calculated as the fifth characteristic parameter according to the first physical marker parameter;
[0030] The extraction process of the sixth characteristic parameter and the seventh characteristic parameter is: predicting the perforation risk of the second target image through a prediction model and generating a perforation confidence and a non-perforation confidence;
[0031] The extraction process of the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter is: obtaining the patient's tumor site, the patient's age and the patient's gender, digitizing the patient's tumor site and the patient's gender, using the digitized patient's tumor site as the eighth characteristic parameter, the Arabic numerals of the patient's age as the ninth characteristic parameter, and the digitized patient's gender as the tenth characteristic parameter.
[0032] In a second aspect, the present invention provides a positioning loss detection model rapid iteration system, comprising:
[0033] A target detection module, used to perform target detection on the ultrasonic endoscopic image using a target detection model to obtain a first detection result and a second detection result;
[0034] A cropping module, used to crop the ultrasonic endoscopic image according to the first detection result to obtain a first target image, and to crop the ultrasonic endoscopic image according to the second detection result to obtain a second target image;
[0035] A semantic segmentation module, used for segmenting the landmark area in the second target image by using a semantic segmentation model to obtain a third target image;
[0036] A physical mark calculation module, used for detecting scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly;
[0037] A feature extraction module, used to extract feature parameters of the ultrasonic endoscopic image, wherein the feature parameters include physical length feature parameters;
[0038] An evaluation and decision-making module is used to evaluate the perforation probability according to the characteristic parameters.
[0039] In a third aspect, the present invention provides a device for risk assessment of lesion resection based on ultrasonic endoscopic images, comprising an image acquisition device and a risk assessment system for lesion resection based on ultrasonic endoscopic images as described in the second aspect, wherein the image acquisition device is connected to an ultrasonic endoscopic detector and the risk assessment system for lesion resection based on ultrasonic endoscopic images, and the image acquisition device is used to obtain an ultrasonic endoscopic detection video and obtain an ultrasonic endoscopic image in the ultrasonic endoscopic detection video.
[0040] In a fourth aspect, the present invention provides a device for risk assessment of lesion resection based on ultrasound endoscopy images, comprising a storage medium and one or more processors, wherein one or more computer programs are stored on the storage medium, and when the one or more computer programs are executed by one or more processors, the method for risk assessment of lesion resection based on ultrasound endoscopy images described in the first aspect is implemented.
[0041] In a fifth aspect, the present invention provides a storage medium storing a computer program. When the computer program is executed by a processor, the steps of the method for risk assessment of lesion resection based on ultrasound endoscopic images described in the first aspect are implemented.
[0042] The present invention is based on the risk assessment method, system and device for lesion resection based on ultrasound endoscopy images, and realizes accurate prediction of the risk of perforation during surgical resection by combining target detection, semantic segmentation and feature extraction technology. The present invention also further improves the accuracy and reliability of perforation risk assessment by combining multiple feature parameters for comprehensive analysis. The present invention realizes accurate conversion from pixel to physical length by determining the physical length corresponding to each pixel, providing a solid foundation for subsequent quantitative analysis.
[0043] The present invention improves the safety of gastrointestinal tumor surgery and reduces the probability of intraoperative complications by accurately assessing the risk of perforation. The present invention realizes efficient processing and analysis of ultrasound images with a high degree of automation, and can provide doctors with an objective and quantitative basis for risk assessment. The present invention provides a new and systematic assessment method, which makes up for the shortcomings of the prior art in predicting perforation risk. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 The present invention is a flow chart of a method for risk assessment of lesion resection based on endoscopic ultrasound images.
[0045] Figure 2 The figure is a schematic diagram of the specific process of the risk assessment method for lesion resection based on endoscopic ultrasound images of the present invention.
[0046] Figure 3 This is a result diagram after S2 is executed in an application example of the method for risk assessment of lesion resection based on endoscopic ultrasound images of the present invention.
[0047] Figure 4 This is a result diagram after S3 is executed in an application example of the method for risk assessment of lesion resection based on endoscopic ultrasound images of the present invention.
[0048] Figure 5 It is a schematic diagram illustrating the position of sub-images in an ultrasound image in S4 in an application example of the method for risk assessment of lesion resection based on ultrasound endoscopy images of the present invention.
[0049] Figure 6 S5.1 is a schematic diagram of the risk assessment method for lesion resection based on endoscopic ultrasound images in an application example of the present invention.
[0050] Figure 7 The structure diagram of the risk assessment system for lesion resection based on endoscopic ultrasound images of the present invention is shown in FIG. DETAILED DESCRIPTION
[0051] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.
[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.
[0053] It should be noted that the descriptions involving "first", "second", etc. in this application are only for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in this field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.
[0054] The lesion resection risk assessment method based on ultrasound endoscopic images is used to fill the gap in existing technologies and provide a lesion resection risk assessment method to provide clinical with reliable perforation risk assessment, help doctors develop safer surgical plans, and reduce surgical risks.
[0055] See also Figure 1 , based on endoscopic ultrasound images, the risk assessment method for lesion resection includes:
[0056] S1. Performing target detection on the ultrasound endoscopy image using a target detection model to obtain a first detection result and a second detection result;
[0057] S2. Cropping the ultrasound endoscopic image according to the first detection result to obtain a first target image, and cropping the ultrasound endoscopic image according to the second detection result to obtain a second target image;
[0058] S3, using a semantic segmentation model to segment the landmark area in the second target image to obtain a third target image;
[0059] S4, detecting the scale line segments in the first target image and calculating the physical length corresponding to each pixel as the first physical marker parameter;
[0060] S5, extracting characteristic parameters of the ultrasound endoscopic image, where the characteristic parameters of the ultrasound endoscopic image include physical length characteristic parameters;
[0061] S6. Evaluate the probability of perforation based on the characteristic parameters of the ultrasound endoscopic image.
[0062] It should be understood that the above method is not limited to the above steps and may vary according to different execution processes. For example, S3 and S4 may be performed simultaneously. Therefore, the above method does not represent or imply that all steps must be performed in this order. Ordinary technicians in this field can transform or change the execution order of the above steps based on the present invention. Some embodiments of the above method are illustrated below.
[0063] See also Figure 2 , Figure 2 Schematic diagram of the specific process of the above method.
[0064] The specific process of the risk assessment method for lesion resection based on endoscopic ultrasound images is as follows:
[0065] S1. Using a pre-trained target detection model based on deep learning, the target detection model can use one of the target detection algorithms such as YOLO, SSD, FASTER-RCNN, etc. The target detection model receives the input ultrasonic endoscopic image, performs automatic target detection on it, and obtains a first detection result and a second detection result.
[0066] S2. The first detection result is the coordinate information of the effective area of the ultrasound endoscopic image. The ultrasound endoscopic image is cropped according to the first detection result to obtain a first target image, that is, the first target image is the image of the effective area in the ultrasound endoscopic image, and the second detection result is the coordinates of the tumor area under ultrasound endoscopy. The image of the tumor and its surrounding area is obtained as the second target image according to the coordinates of the tumor area. Figure 3 , Figure 3 The medium-large green box is the first target image, and the small one is the second target image.
[0067] Please see again Figure 3According to the preset expansion coefficient, the tumor coordinate area is expanded to obtain new expansion coordinates, so that the area within the new expansion coordinates contains the image information of the tumor and its surrounding mucosal layers. The expansion coefficient adopts the preset method. The expansion coefficient used in the present invention is 1 / 2 of the width and height of the target image. The upper and lower parts of the second detection result are expanded by 1 / 2 of the height, and the left and right parts are expanded by 1 / 2 of the width, respectively, to obtain the tumor coordinate area and its surrounding area. This area is used as the pre-acquisition area, and the ultrasound endoscope image is cropped to obtain the second target image. The second target image is the image of the tumor and its surrounding area in the ultrasound endoscope image. See Figure 4 .
[0068] S3, using a semantic segmentation model to segment the landmark areas in the second target image to obtain a third target image. The landmark areas include background (background area), submucosal tumor area, mucosal layer area, mucosal muscularis area, submucosal layer area, muscularis propria area, and serosa area.
[0069] Specifically, a semantic segmentation model may be used to segment and distinguish and mark landmark areas to obtain a third target image.
[0070] Here, the semantic segmentation network is a pre-trained model, and the semantic segmentation model is obtained by training the semantic segmentation network using a pre-constructed segmentation mask dataset. The semantic segmentation network model can use UNet, SegNet, Deeplab, FCN, etc. The segmentation targets of the semantic segmentation model include: background, submucosal tumor area, mucosal layer area, muscularis mucosa area, submucosal layer area, muscularis propria area, and serosa layer area. It can be understood that the second target image will have several types of background, submucosal tumor area, mucosal layer area, muscularis mucosa area, submucosal layer area, muscularis propria area, and serosa layer area. The semantic segmentation network model can mark each area with a set color, and the unmarked area can be set as the background area. The image after segmentation and distinguishing and marking each area is the third target image, which can be seen in Figure 4 . Distinguishing marks are not limited to this type.
[0071] S4. Use an image smoothing algorithm and an edge extraction image processing algorithm to process the first target image, then detect the scale segments in the processed first target image, determine the pixel position information of the scale segments, calculate the pixel distance between adjacent scale segments, and calculate the physical length corresponding to each pixel as the first physical marker parameter based on the actual physical distance between adjacent scale segments.
[0072] See also Figure 5, the scale line segments are scale lines in the ultrasound endoscopic image. The first target image is a rectangular frame in the ultrasound endoscopic image overall interface after the detection date, time, FR, DG, MI, TIS and other parameters are cut out, but the first target image includes the scale line segments in the ultrasound endoscopic image. In the art, the scale line segments are located on at least one of the upper and lower sides of the first target image as scale lines on the horizontal coordinate axis; the scale line segments are also located on at least one of the left and right sides as scale lines on the vertical coordinate axis. Figure 5 In the figure, the scale lines are located above and to the right of the first test result of the ultrasound endoscopic image. Figure 5 The red frame area is the sub-image.
[0073] Usually, there are long scale lines and short scale lines in the ultrasonic endoscopy image, that is, there is a short scale line in the middle of two long scale lines, and there is a long scale line in the middle of two short scale lines.
[0074] Specifically, S4 is: cropping the first target image to obtain a sub-image with scale line segments, processing the sub-image using an image smoothing processing algorithm and an edge extraction image processing algorithm, detecting straight line segments in the processed sub-image, classifying the straight line segments obtained by the clustering algorithm to determine the pixel position information of the straight line segments, calculating the pixel distance between adjacent straight line segments in the same direction, and calculating the physical length corresponding to each pixel as the first physical marker parameter according to the actual physical distance between the scale line segments. The straight line segment is usually a scale line segment.
[0075] The S4 includes:
[0076] S4.1, image cropping: according to the prior information, the first target image is cropped to obtain the first sub-image and the second sub-image. The prior information is the position information of the scale line in the first target image, and the prior information can be determined by human visual inspection and analysis. For example, the prior information is: the first target image has scale lines of the horizontal axis at the top and bottom, and scale lines of the vertical axis on the right, and they are all within 50 pixels, then the specific operation of S4.1 is: crop 50 pixels from the top and bottom of the first target image to obtain the first sub-image, and crop 50 pixels from the rightmost to the left to obtain the second sub-image. It can be understood that the first sub-image and the second sub-image both have scale line segments.
[0077] S4.2. Perform the following processing on the first sub-image and the second sub-image respectively to obtain the first processed sub-image and the second processed sub-image in a one-to-one correspondence: convert both images into grayscale images, apply a 5×5 Gaussian filter to smooth the grayscale images, use the Canny edge detection operator to perform edge detection processing, and obtain a binary image with a more refined representation.
[0078] S4.3. Detect the straight line segments in the first target image after the first processed sub-image and the second processed sub-image are processed, classify the straight line segments to obtain the straight line segment positions, calculate the pixel distances between adjacent straight line segments based on the straight line segment positions, and calculate the physical length corresponding to each pixel as the first physical marker parameter according to the actual physical distance between the scale lines.
[0079] S4.31, using Hough line transform to detect straight line segments in the edge image for the first processed sub-image and the second processed sub-image. To ensure that only straight line segments are detected, the present invention sets a minimum line segment length (such as 10 pixels) and a maximum line segment gap (such as 5 pixels), and filters out line segments with horizontal, vertical, nearly horizontal, and nearly horizontal and vertical directions. Here, for the first processed sub-image, vertical and nearly vertical line segments are filtered out, and for the second processed sub-image, horizontal and nearly horizontal line segments are filtered out.
[0080] S4.32, Cluster analysis: Through line segment cluster analysis, line segments with similar positions and consistent directions are classified into one category, and the clustering result is the pixel position information of each straight line segment. Line segments close to horizontal and horizontal are in the same direction, and line segments close to vertical and vertical are in the same direction.
[0081] Here, the line segment cluster analysis is classified by length and position. The line segments in the same class not only have the same length (which may be close), but also have almost the same horizontal (or vertical) coordinates at one end and almost the same horizontal (or vertical) coordinates at the other end.
[0082] It can be understood that they are classified into at least two categories, one is short scale lines, and the other is long scale lines. The horizontal coordinates or vertical coordinates in the position coordinates of all long scale lines are almost the same, and the horizontal coordinates or vertical coordinates in the position coordinates of all short scale lines are almost the same. It can be understood that there may be other straight lines in the sub-image, which will also form several categories separately according to the situation. It can be understood that the scale line category may have other straight line segments besides the scale line segments.
[0083] Preferably, S4.32 also includes the step of eliminating some clustering result categories according to the position coordinates of the straight line segments in the clustering results, that is, eliminating the categories that are not scale lines. The basis for elimination may be quantity, uniformity of spacing, etc.
[0084] S4.33. Calculate the pixel distance between adjacent straight line segments:
[0085] For the first sub-image: based on the pixel position information of the vertical straight line segments obtained by clustering, select adjacent straight line segments and calculate the pixel distance D_pixels between their center points. For the second sub-image: based on the pixel position information of the horizontal straight line segments obtained by clustering, select adjacent straight line segments and calculate the pixel distance D_pixels between their center points.
[0086] To improve accuracy, the pixel distances of multiple adjacent straight line segments are fully calculated from left to right and / or from top to bottom, and the two-part method only calculates the local area, and the average value D_avg_pixels is taken after calculation. Preferably, not only the average value but also the mode is considered, and the pixel distances of adjacent straight line segments are determined by comprehensive analysis.
[0087] In a preferred embodiment, in this step, only the pixel distances between adjacent long straight line segments (long scale lines) are calculated, and the pixel distances between adjacent short straight line segments are not calculated.
[0088] S4.34, Physical length calculation: Using the known physical scale length (the actual physical distance between the scale lines, such as: the actual distance between two horizontal long scale lines is 10mm, the physical scale length is generally set manually and obtained in advance), calculate the physical length corresponding to each pixel L_per_pixel = 10mm / D_avg_pixels. This value is used as the first physical marker parameter in subsequent measurements.
[0089] It can be understood that if the length and width of each pixel are equal, only the first sub-image or the second sub-image may be obtained, or only the horizontal or vertical straight line segments may be screened out.
[0090] S5, extracting characteristic parameters according to the third target image, the second target image, etc., including physical length characteristic parameters;
[0091] S5.1. Extract the first characteristic parameter, the second characteristic parameter and the third characteristic parameter from the third target image, wherein the first characteristic parameter and the second characteristic parameter are physical length characteristic parameters: Generate the first, second and third characteristic parameters by calculating the vertical distance from the mucosal layer area to the top of the submucosal tumor area and the distance from the mucosal layer area to the serosal layer area in the third target image. For details on the confirmation method of the first, second and third characteristic parameters, see Figure 6 for:
[0092] First, rotation correction is performed: for the mucosal layer area and the submucosal tumor area in the third target image, the principal component analysis (PCA) algorithm is used to calculate their principal axis directions. According to the principal axis directions, the rotation angle θ is calculated, and then the third target image is rotated according to the rotation angle θ to obtain the preprocessed third target image, that is, the rotated third target image;
[0093] Secondly, determine the vertex: on the pre-processed third target image, start from the upper boundary of the mucosal layer area, record the intersection with the horizontal line as P0, scan downward line by line, find the first intersection point of the submucosal tumor area, record it as P1. Similarly, scan the last intersection point of the serous layer area, record it as P2. The above method can be performed after extracting the mucosal layer area and the submucosal tumor area in the third target image.
[0094] Finally, see Figure 6 , perform vertical distance calculation: calculate the vertical distance between P0 and P1 to obtain L1; calculate the vertical distance between P1 and P2 to obtain L2. Combined with the first physical marker parameter, convert the L1 and L2 pixel distances into physical distances to obtain the first characteristic parameter and the second characteristic parameter; at the same time, calculate the ratio of L1 to L2 and record it as the third characteristic parameter.
[0095] The first characteristic parameter is called the physical distance between the mucosal layer area and the submucosal tumor area, the second characteristic parameter is called the physical distance between the submucosal tumor area and the serosal layer area, the second characteristic parameter includes the physical distance of the serosal layer area, and the third characteristic parameter is called the ratio of the first characteristic parameter to the second characteristic parameter.
[0096] S5.2, extracting the fourth characteristic parameter and the fifth characteristic parameter: generating them by calculation based on the physical length of the long and short axes of the tumor contour. This process uses the physical marker parameters to convert the pixel length into the actual physical length.
[0097] Specifically:
[0098] On the preprocessed third target image, the minimum enclosing rectangle algorithm is used to determine the long and short axes of the submucosal tumor area. Long axis calculation: The diagonal length of the minimum enclosing rectangle is calculated as the long axis of the tumor. Short axis calculation: The short axis length is measured in the direction perpendicular to the long axis.
[0099] Physical length conversion: The long axis and short axis pixel lengths of the tumor calculated above are converted into the physical long axis length and physical short axis length of the tumor through the first physical marker parameter obtained previously, and the fourth and fifth characteristic parameters are obtained. The fourth characteristic parameter is the physical length of the long axis of the submucosal tumor area, and the fifth characteristic parameter is the physical length of the short axis of the submucosal tumor area. The fourth and fifth characteristic parameters are physical length characteristic parameters.
[0100] S5.3. Extract the sixth characteristic parameter and the seventh characteristic parameter: Based on the labeled data set, a CNN model (convolutional neural network model such as ResNet) is constructed and trained. The prediction model is used to predict the perforation risk of the second target image and generate perforation confidence and non-perforation confidence to obtain the sixth characteristic parameter and the seventh characteristic parameter.
[0101] S5.4. Obtain the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter: obtain the patient information corresponding to the ultrasonic endoscopic image, the site of the patient's tumor (for example, one of the upper gastric body, middle gastric body, lower gastric body, fundus, gastric antrum, and gastric angle), the patient's age, and gender, which can be digitized. For example, the patient's gender is defined as 0 or 1, the upper gastric body, middle gastric body, lower gastric body, fundus, gastric antrum, and gastric angle are defined as numbers 2 to 9, respectively, the (digitized) site of the patient's tumor is used as the eighth characteristic parameter, the Arabic numerals of the patient's age are used as the ninth characteristic parameter, and the (digitized) patient's gender is used as the tenth characteristic parameter.
[0102] S6. Evaluate the risk of perforation for each case based on the first to tenth characteristic parameters;
[0103] For the perforation risk of each case: use the pre-trained evaluation decision module to input the above feature parameters into the decision model (such as XGBoost, decision tree, random forest, SVM, etc.) to output the perforation risk probability.
[0104] The risk assessment method for lesion resection based on endoscopic ultrasound images of the present invention combines target detection, semantic segmentation and feature extraction technology, not only provides an assessment method for the risk of resection perforation, fills the medical gap, provides reliable perforation risk assessment for clinicians, helps doctors formulate safer surgical plans and reduce surgical risks; but also realizes accurate prediction of perforation risk during surgical resection. The present invention further improves the accuracy and reliability of perforation risk assessment by combining multiple feature parameters for comprehensive analysis. The present invention realizes accurate conversion from pixels to physical length by determining the first physical marker parameter, providing a solid foundation for subsequent quantitative analysis. The present invention improves the safety of digestive tract tumor surgery, reduces the probability of intraoperative complications by accurately assessing the risk of perforation. The present invention realizes efficient processing and analysis of ultrasound images, has a high degree of automation, and can provide doctors with objective and quantitative risk assessment basis. The present invention provides a new and systematic assessment method, which makes up for the shortcomings of the prior art in perforation risk prediction.
[0105] See also Figure 7 The present invention provides a risk assessment system for lesion resection based on endoscopic ultrasound images, comprising:
[0106] The target detection module 10 is used to perform target detection on the ultrasonic endoscopic image using the target detection model to obtain a first detection result and a second detection result;
[0107] A cropping module 20, configured to crop the ultrasound endoscopic image according to the first detection result to obtain a first target image, and crop the ultrasound endoscopic image according to the second detection result to obtain a second target image;
[0108] A semantic segmentation module 30, configured to segment the landmark area in the second target image using a semantic segmentation model to obtain a third target image;
[0109] A physical mark calculation module 40, for detecting the scale line segments in the first target image and calculating the physical length corresponding to each pixel as a first physical mark parameter;
[0110] A feature extraction module 50 is used to extract feature parameters of the ultrasonic endoscopic image, wherein the feature parameters include physical length feature parameters;
[0111] The evaluation and decision module 60 is used to evaluate the perforation probability according to the characteristic parameters.
[0112] The first target image includes scale line segments in the ultrasonic endoscopy image; the physical marker calculation module 40 is specifically used to crop the first target image to obtain a sub-image with scale line segments, process the sub-image using an image smoothing algorithm and an edge extraction image processing algorithm, and then detect the straight line segments in the processed sub-image, determine the pixel position information of the straight line segments, calculate the pixel distance between adjacent straight line segments, and calculate the physical length corresponding to each pixel based on the actual physical distance of the scale line spacing.
[0113] Furthermore, the physical mark calculation module 40 includes:
[0114] A first cropping unit, used for cropping the first target image to obtain a first sub-image and a second sub-image, wherein the first sub-image has scale marks on the horizontal axis, and the second sub-image has scale marks on the vertical axis;
[0115] a conversion unit, configured to convert the first sub-image and the second sub-image into grayscale images,
[0116] A smoothing processing unit, used for smoothing the grayscale image;
[0117] An edge detection unit is used for performing edge detection processing on the image smoothed by the smoothing processing unit to obtain a binary image, that is, to obtain a first processed sub-image and a second processed sub-image;
[0118] A straight line detection unit, used to detect straight line segments in the edge image of the first processed sub-image and the second processed sub-image;
[0119] A clustering analysis module is used to perform clustering analysis on straight line segments to obtain pixel position information of each straight line segment;
[0120] A first calculation unit is used to calculate the pixel distance between adjacent straight line segments in the same class in line segment cluster analysis;
[0121] The second calculation unit is used to calculate the physical length corresponding to each pixel by using the actual physical distance of the known scale line spacing, so as to obtain the first physical mark parameter.
[0122] The semantic segmentation module 30 is specifically used to segment the landmark areas in the second target image and perform distinguishing marks.
[0123] The feature extraction module 50 comprises:
[0124] A first feature extraction unit is used to extract the first feature parameter, the second feature parameter, and the third feature, specifically, to calculate the main axis direction of the mucosal layer area and the submucosal tumor area, to rotate the third target image according to the main axis direction to obtain the rotated third target image, to determine the distance between the mucosal layer area and the submucosal tumor area in the vertical direction, to determine the distance between the submucosal tumor area and the bottom end of the serosal layer area in the vertical direction, to calculate the first feature parameter and the second feature parameter according to the first physical marker parameter, and to calculate the third feature parameter;
[0125] A second feature extraction unit is used to extract the fourth feature parameter and the fifth feature parameter, and is specifically used to use a minimum circumscribed rectangle algorithm to determine the long axis and the short axis of the submucosal tumor area on the rotated third target image, wherein the short axis is perpendicular to the long axis, and is used to calculate the long axis and the short axis in a one-to-one correspondence according to the first physical marker parameter as the fourth feature parameter and the fifth feature parameter;
[0126] a third feature extraction unit, used to extract the sixth feature parameter and the seventh feature parameter, specifically used to predict the perforation risk of the second target image through a prediction model and generate a perforation confidence and a non-perforation confidence;
[0127] The fourth feature extraction unit is used to extract the eighth feature parameter, the ninth feature parameter and the tenth feature parameter, and is specifically used to obtain the patient's tumor site, the patient's age and the patient's gender, and is used to digitize the patient's tumor site and the patient's gender, using the digitized patient's tumor site as the eighth feature parameter and the digitized patient's gender as the tenth feature parameter.
[0128] When the lesion resection risk assessment system based on endoscopic ultrasound images is implemented, reference may be made to the lesion resection risk assessment system implemented by the lesion resection risk assessment method based on endoscopic ultrasound images in any of the above embodiments, and the specific implementation steps will not be repeated here.
[0129] The present invention provides a risk assessment device for lesion resection based on ultrasonic endoscopic images, comprising an image acquisition device and the risk assessment system for lesion resection based on ultrasonic endoscopic images, wherein the image acquisition device is connected to an ultrasonic endoscopic detector and the risk assessment system for lesion resection based on ultrasonic endoscopic images, and the image acquisition device is used to obtain an ultrasonic endoscopic detection video and obtain an ultrasonic endoscopic image in the ultrasonic endoscopic detection video.
[0130] Furthermore, the device also includes a display for displaying the evaluation results of the lesion resection risk evaluation system based on the ultrasonic endoscopic image.
[0131] The present invention also provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the method for risk assessment of lesion resection based on ultrasonic endoscopic images.
[0132] The processor may be a CPU, or other general-purpose processors, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0133] The memory may include various types of storage units, such as system memory, read-only memory, and permanent storage. In addition, the memory may include any combination of computer-readable storage media, such as semiconductor memory chips, magnetic disks, and optical disks.
[0134] The memory stores executable codes, and when the executable codes are processed by the processor, the processor can execute part or all of the methods described above.
[0135] The technicians in the field to which the present invention belongs can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0136] It should be noted that in the above embodiments, the description of each embodiment has its own emphasis, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0137] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, etc. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0138] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0139] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for assessing the risk of lesion resection based on endoscopic ultrasound images, characterized in that: include: Using the target detection model to perform target detection on the ultrasound endoscopy image to obtain a first detection result and a second detection result; the first detection result is the coordinate information of the effective area of the ultrasound endoscopy image, and the second detection result is the coordinates of the tumor area under the ultrasound endoscopy; The first target image is obtained by cropping the ultrasound endoscopic image according to the first detection result, and the second target image is obtained by cropping the ultrasound endoscopic image according to the second detection result; the first target image is an image of a valid area in the ultrasound endoscopic image, and the second target image is an image of the tumor and its surrounding area in the ultrasound endoscopic image; Using a semantic segmentation model to segment the landmark area in the second target image to obtain a third target image; Detecting scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly; Extracting feature parameters based on the third target image and the second target image, the feature parameters including physical length feature parameters, the physical length feature parameters being obtained by calculating using the physical length corresponding to each pixel; The perforation probability is evaluated according to the characteristic parameters.
2. The method for risk assessment of lesion resection based on endoscopic ultrasound images according to claim 1, characterized in that: The first target image includes scale line segments in the ultrasound endoscopy image; The method of detecting the scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly specifically includes: cropping the first target image to obtain a sub-image with scale line segments, processing the sub-image using an image smoothing processing algorithm and an edge extraction image processing algorithm, then detecting the straight line segments in the processed sub-image, determining the pixel position information of the straight line segments, calculating the pixel distance between adjacent straight line segments, and calculating the physical length corresponding to each pixel based on the actual physical distance between the scale line segments.
3. The method for risk assessment of lesion resection based on endoscopic ultrasound images as claimed in claim 2, characterized in that: The detecting of the scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly specifically includes: Cropping the first target image to obtain a first sub-image and a second sub-image, wherein the first sub-image has scale marks on the horizontal axis, and the second sub-image has scale marks on the vertical axis; The first sub-image and the second sub-image are respectively processed as follows to obtain a first processed sub-image and a second processed sub-image in a one-to-one correspondence: both images are converted into grayscale images, the grayscale images are smoothed, and then edge detection is performed to obtain binary images; Detecting straight line segments in the edge images of the first processed sub-image and the second processed sub-image; The pixel position information of each straight line segment is obtained by performing line segment clustering analysis on the straight line segment; Calculate the pixel distance between adjacent straight line segments in the same class of line segment cluster analysis; The physical length corresponding to each pixel is calculated using the known actual physical distance of the scale line spacing, thereby obtaining the first physical marker parameter.
4. The method for risk assessment of lesion resection based on endoscopic ultrasound images according to claim 1, characterized in that: The semantic segmentation model is used to segment the landmark areas in the second target image, specifically: segment the landmark areas in the second target image and distinguish and mark them, wherein the landmark areas include background, submucosal tumor area, mucosal layer area, mucosal muscular layer area, submucosal layer area, muscular layer propria area, and serosal layer area.
5. The method for risk assessment of lesion resection based on endoscopic ultrasound images according to claim 4, characterized in that: The characteristic parameters include part or all of the first characteristic parameter, the second characteristic parameter, the third characteristic parameter, the fourth characteristic parameter, the fifth characteristic parameter, the sixth characteristic parameter, the seventh characteristic parameter, the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter; The first characteristic parameter is the physical distance between the mucosal layer area and the submucosal tumor area, the second characteristic parameter is the physical distance between the submucosal tumor area and the serosal layer area, the second characteristic parameter includes the physical distance of the serosal layer area, and the third characteristic parameter is the ratio of the first characteristic parameter to the second characteristic parameter; The fourth characteristic parameter is the physical length of the submucosal tumor area, and the fifth characteristic parameter is the physical length of the short axis of the submucosal tumor area; The sixth characteristic parameter is the perforation confidence, and the seventh characteristic parameter is the non-perforation confidence; The eighth characteristic parameter is the site of tumor occurrence in the patient, the ninth characteristic parameter is the patient's age, and the tenth characteristic parameter is the patient's gender.
6. The method for risk assessment of lesion resection based on endoscopic ultrasound images according to claim 5, characterized in that: The extraction process of the first characteristic parameter, the second characteristic parameter and the third characteristic parameter is as follows: Calculating the main axis directions of the mucosal layer region and the submucosal tumor region, and rotating the third target image according to the main axis directions to obtain a rotated third target image; determining the distance between the mucosal layer region and the submucosal tumor region along the vertical direction, determining the distance between the submucosal tumor region and the bottom end of the serous layer region along the vertical direction, calculating the first characteristic parameter and the second characteristic parameter according to the first physical marker parameter, and calculating the third characteristic parameter; The extraction process of the fourth characteristic parameter and the fifth characteristic parameter is as follows: The minimum circumscribed rectangle algorithm is used to determine the long axis and short axis of the submucosal tumor region of the rotated third target image, wherein the short axis is perpendicular to the long axis, and the physical length of the long axis is calculated as the fourth characteristic parameter according to the first physical marker parameter, and the physical length of the short axis is calculated as the fifth characteristic parameter according to the first physical marker parameter; The extraction process of the sixth characteristic parameter and the seventh characteristic parameter is: predicting the perforation risk of the second target image through a prediction model and generating a perforation confidence and a non-perforation confidence; The extraction process of the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter is: obtaining the patient's tumor site, the patient's age and the patient's gender, digitizing the patient's tumor site and the patient's gender, using the digitized patient's tumor site as the eighth characteristic parameter, the Arabic numerals of the patient's age as the ninth characteristic parameter, and the digitized patient's gender as the tenth characteristic parameter.
7. The risk assessment system for lesion resection based on endoscopic ultrasound images is characterized by: include: A target detection module, used to perform target detection on the ultrasound endoscopy image using a target detection model to obtain a first detection result and a second detection result, wherein the first detection result is the coordinate information of the effective area of the ultrasound endoscopy image, and the second detection result is the coordinates of the tumor area under the ultrasound endoscopy; a cropping module, configured to crop the endoscopic ultrasound image according to the first detection result to obtain a first target image, and crop the endoscopic ultrasound image according to the second detection result to obtain a second target image, wherein the first target image is an image of a valid area in the endoscopic ultrasound image, and the second target image is an image of the tumor and its surrounding area in the endoscopic ultrasound image; A semantic segmentation module, used for segmenting the landmark area in the second target image by using a semantic segmentation model to obtain a third target image; A physical mark calculation module, used for detecting scale line segments in the first target image and calculating the physical length corresponding to each pixel accordingly; A feature extraction module, used to extract feature parameters based on the third target image and the second target image, wherein the feature parameters include physical length feature parameters, and the physical length feature parameters are obtained by calculating the physical length corresponding to each pixel; An evaluation and decision-making module is used to evaluate the perforation probability according to the characteristic parameters.
8. A device for assessing the risk of lesion resection based on endoscopic ultrasound images, characterized in that: It comprises an image acquisition device and the risk assessment system for lesion resection based on ultrasonic endoscopic images as described in claim 7, wherein the image acquisition device is connected to an ultrasonic endoscopic detector and the risk assessment system for lesion resection based on ultrasonic endoscopic images, and the image acquisition device is used to obtain an ultrasonic endoscopic detection video and obtain an ultrasonic endoscopic image in the ultrasonic endoscopic detection video.
9. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for risk assessment of lesion resection based on endoscopic ultrasound images as described in any one of claims 1 to 6 are implemented.
