Endoscopic ultrasound image-based lesion excision risk evaluation method, system and apparatus

Abstract

Disclosed in the present invention are an endoscopic ultrasonic image-based lesion excision risk evaluation method, system and apparatus, the method comprising: using an object detection model to perform object detection on an endoscopic ultrasonic image to obtain a first detection result and a second detection result; on the basis of the first detection result, cropping the endoscopic ultrasonic image to obtain a first target image and, on the basis of the second detection result, cropping the endoscopic ultrasonic image to obtain a second target image; using a semantic segmentation model to segment a landmark region in the second target image to obtain a third target image; detecting scale line segments in the first target image and, on the basis of the scale line segments, calculating a physical length corresponding to each pixel; extracting feature parameters of the endoscopic ultrasonic image, the feature parameters comprising a physical length feature parameter; and, on the basis of the feature parameters, evaluating a perforation probability. The present invention accurately predicts perforation risks during surgical excision, provides a solid basis for subsequent quantitative analysis, and improves the safety of gastrointestinal tumor surgery.

Description

Method, system and device for evaluating risk of lesion resection based on endoscopic ultrasound image

[0001] Cross-reference to Related Applications

[0002] The present application claims priority to the Chinese patent application No. 202411161591.0, filed on August 23, 2024, and entitled "Method, system and device for evaluating risk of lesion resection based on endoscopic ultrasound image", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of EUS image processing, and in particular to a method, system and device for evaluating risk of lesion resection based on endoscopic ultrasound image, and a storage medium. BACKGROUND

[0004] Currently, endoscopic ultrasound (EUS) has become an important tool for diagnosing digestive tract lesions, especially in assessing the origin level and invasion depth of digestive tract tumors. Existing technologies have already used artificial intelligence, especially convolutional neural networks (CNN), to assist doctors in identifying and classifying lesion areas.

[0005] For example, the patent with publication number CN115240019A discloses a method for identifying the origin layer of digestive tract tumors based on convolutional neural networks, which identifies the origin level of digestive tract tumors by setting classification indicators. The patent with publication number CN117218127A discloses an endoscopic ultrasound assisted monitoring system that can automatically identify and segment lesion areas and generate lesion marked ultrasound images to help doctors make more accurate diagnoses. The patent with publication number CN116823695A proposes a gastric submucosal protrusion classification system based on multi-modal image data and deep learning methods, which realizes the classification and screening of gastric submucosal protrusions by constructing multiple AI models.

[0006] Although these existing technologies can assist doctors in classifying and locating lesions to some extent, in actual treatment, especially in the process of lesion resection, there is still a key challenge of how to accurately evaluate the risk of perforation that may occur during the resection process. Current technical means mainly focus on the diagnosis and classification of lesions, and lack objective evaluation criteria for the risk of resection perforation, which has become a problem that needs to be solved in clinical practice. Therefore, it is necessary to develop a method for evaluating the risk of lesion resection based on endoscopic ultrasound images. SUMMARY

[0007] In view of the above problems, the present application provides an endoscopic ultrasound image-based lesion resection risk assessment method, system and device, aiming to provide an accurate assessment method for resection perforation risk, fill in the medical blank, and provide reliable perforation risk assessment for clinical practice, help doctors develop safer surgical plans, and reduce surgical risk.

[0008] The technical solution adopted by the present application to solve the technical problems is as follows:

[0009] In a first aspect, the present application provides an endoscopic ultrasound image-based lesion resection risk assessment method, comprising:

[0010] A target detection model is used to perform target detection on the endoscopic ultrasound image to obtain first and second detection results;

[0011] The endoscopic ultrasound image is cropped according to the first detection result to obtain a first target image, and the endoscopic ultrasound image is cropped according to the second detection result to obtain a second target image;

[0012] A semantic segmentation model is used to segment a landmark region in the second target image to obtain a third target image;

[0013] Scale lines in the first target image are detected and the physical length corresponding to each pixel is calculated accordingly;

[0014] Feature parameters of the endoscopic ultrasound image are extracted, and the feature parameters include physical length feature parameters;

[0015] The perforation probability is evaluated according to the feature parameters.

[0016] In a preferred embodiment, the first detection result is coordinate information of the effective area of the endoscopic ultrasound image, and the first target image is an image of the effective area of the endoscopic ultrasound image; the second detection result is the coordinate of the tumor region under the endoscope; and the second target image is an image of the tumor and its surrounding area in the endoscopic ultrasound image.

[0017] In a preferred embodiment, the first target image includes scale lines in the endoscopic ultrasound image; and the detection of scale lines in the first target image and the calculation of the physical length corresponding to each pixel specifically include: cropping the first target image to obtain a sub-image with scale lines, processing the sub-image using an image smoothing algorithm and an edge extraction image processing algorithm, then detecting straight line segments in the processed sub-image, determining the pixel position information of the straight line segments, calculating the pixel distance of adjacent straight line segments, and calculating the physical length corresponding to each pixel according to the actual physical distance between the scale lines.

[0018] In a preferred embodiment, the detection of scale lines in the first target image and the calculation of the physical length corresponding to each pixel specifically include:

[0019] cropping the first target image to obtain a first sub-image and a second sub-image, the first sub-image having a scale line of an abscissa axis, and the second sub-image having a scale line of an ordinate axis;

[0020] respectively obtaining a first processed sub-image and a second processed sub-image by performing the following processing on the first sub-image and the second sub-image respectively: converting the images into gray-scale images, performing smoothing processing on the gray-scale images, and then performing edge detection processing to obtain binary images;

[0021] detecting straight line segments in the edge images of the first processed sub-image and the second processed sub-image;

[0022] obtaining pixel position information of each straight line segment by performing line segment clustering analysis on the straight line segments;

[0023] calculating pixel distances between adjacent straight line segments in the same class of the line segment clustering analysis;

[0024] calculating a physical length corresponding to each pixel by using a known actual physical distance between the scale lines, i.e., obtaining a first physical marker parameter.

[0025] In a preferred embodiment, the semantic segmentation model is used to segment the marker region in the second target image, specifically: segmenting the marker region in the second target image and marking the segmented regions, wherein the marker region includes a background, a submucosal tumor region, a mucosa layer region, a muscularis mucosa layer region, a submucosa layer region, an intrinsic muscle layer region, and a serosa layer region.

[0026] In a preferred embodiment, the feature parameters include some or all of a first feature parameter, a second feature parameter, a third feature parameter, a fourth feature parameter, a fifth feature parameter, a sixth feature parameter, a seventh feature parameter, an eighth feature parameter, a ninth feature parameter, and a tenth feature parameter.

[0027] The first feature parameter is a physical distance between the mucosa layer region and the submucosal tumor region, the second feature parameter is a physical distance between the submucosal tumor region and the serosa layer region, the second feature parameter includes a physical distance of the serosa layer region, the third feature parameter is a ratio of the first feature parameter to the second feature parameter, the fourth feature parameter is a physical length of the submucosal tumor region, the fifth feature parameter is a physical length of a short axis of the submucosal tumor region, the sixth feature parameter is a perforation confidence, the seventh feature parameter is a non-perforation confidence, the eighth feature parameter is a tumor occurrence site of a patient, the ninth feature parameter is a patient age, and the tenth feature parameter is a patient gender.

[0028] In a preferred embodiment, the extraction processes of the first feature parameter, the second feature parameter, and the third feature parameter are as follows:

[0029] The principal axis direction of the mucosa layer region and the submucosa tumor region is calculated, and the third target image is rotated according to the principal axis direction, to obtain a rotated third target image; the distance of the mucosa layer region and the submucosa tumor region in the vertical direction is determined, the distance of the submucosa tumor region and the bottom end of the serosa layer region in the vertical direction is determined, the first characteristic parameter and the second characteristic parameter are calculated according to the first physical mark parameter, and the third characteristic parameter is calculated;

[0030] The extraction process of the fourth characteristic parameter and the fifth characteristic parameter is as follows:

[0031] The long axis and the short axis of the submucosa tumor region are determined by using the minimum circumscribed rectangle algorithm on the rotated third target image, the short axis is perpendicular to the long axis, the physical length of the long axis is calculated as the fourth characteristic parameter according to the first physical mark parameter, and the physical length of the short axis is calculated as the fifth characteristic parameter according to the first physical mark parameter;

[0032] The extraction process of the sixth characteristic parameter and the seventh characteristic parameter is as follows: the perforation risk of the second target image is predicted by using a prediction model, and a perforation confidence and a non-perforation confidence are generated;

[0033] The extraction process of the eighth characteristic parameter, the ninth characteristic parameter and the tenth characteristic parameter is as follows: the patient's tumor occurrence site, the patient's age and the patient's gender are obtained, the patient's tumor occurrence site and the patient's gender are digitally processed, the digital patient's tumor occurrence site is taken as the eighth characteristic parameter, the Arabic numerals of the patient's age are taken as the ninth characteristic parameter, and the digital patient's gender is taken as the tenth characteristic parameter.

[0034] In a second aspect, the present application provides a positioning loss detection model rapid iteration system, comprising:

[0035] A target detection module is configured to perform target detection on the endoscopic ultrasound image by using a target detection model to obtain a first detection result and a second detection result;

[0036] A cropping module is configured to crop the endoscopic ultrasound image according to the first detection result to obtain a first target image, and crop the endoscopic ultrasound image according to the second detection result to obtain a second target image;

[0037] A semantic segmentation module is configured to segment a landmark region in the second target image by using a semantic segmentation model to obtain a third target image;

[0038] A physical mark calculation module is configured to detect scale line segments in the first target image and calculate the physical length corresponding to each pixel according to the scale line segments;

[0039] A feature extraction module is configured to extract characteristic parameters of the endoscopic ultrasound image, wherein the characteristic parameters include physical length characteristic parameters;

[0040] an evaluation decision module configured to evaluate the perforation probability according to the feature parameters.

[0041] In a third aspect, the present application provides an apparatus for evaluating the risk of lesion resection based on an endoscopic ultrasound image, comprising an image obtaining device and the system for evaluating the risk of lesion resection based on an endoscopic ultrasound image as described in the second aspect, wherein the image obtaining device is connected to an endoscopic ultrasound detector and the system for evaluating the risk of lesion resection based on an endoscopic ultrasound image, and the image obtaining device is configured to obtain an endoscopic ultrasound detection video and obtain an endoscopic ultrasound image in the endoscopic ultrasound detection video.

[0042] In a fourth aspect, the present application provides an apparatus for evaluating the risk of lesion resection based on an endoscopic ultrasound image, comprising a storage medium and one or more processors, wherein the storage medium stores one or more computer programs, and when the one or more computer programs are executed by the one or more processors, the method for evaluating the risk of lesion resection based on an endoscopic ultrasound image as described in the first aspect is implemented.

[0043] In a fifth aspect, the present application provides a storage medium storing a computer program, and when the computer program is executed by a processor, the steps of the method for evaluating the risk of lesion resection based on an endoscopic ultrasound image as described in the first aspect are implemented.

[0044] The method, system and apparatus for evaluating the risk of lesion resection based on an endoscopic ultrasound image of the present application realize accurate prediction of the risk of perforation during surgical resection by combining target detection, semantic segmentation and feature extraction technology. The present application further improves the accuracy and reliability of the risk of perforation evaluation by combining a variety of feature parameters for comprehensive analysis. The present application realizes accurate conversion from pixels to physical length by determining the corresponding physical length of each pixel, which provides a solid foundation for subsequent quantitative analysis.

[0045] The present application improves the safety of gastrointestinal tumor surgery by accurately evaluating the risk of perforation, thereby reducing the probability of intraoperative complications. The present application realizes efficient processing and analysis of ultrasound images, has high automation, and can provide objective and quantitative risk evaluation basis for doctors. The present application provides a new and systematic evaluation method, which makes up for the shortcomings of the prior art in perforation risk prediction. BRIEF DESCRIPTION OF DRAWINGS

[0046] Fig. 1 is a flowchart of the method for evaluating the risk of lesion resection based on an endoscopic ultrasound image of the present application.

[0047] Fig. 2 is a schematic diagram of the specific process of the method for evaluating the risk of lesion resection based on an endoscopic ultrasound image of the present application.

[0048] Fig. 3 is a result diagram after S2 is executed in an application example of the method for evaluating the risk of lesion resection based on an endoscopic ultrasound image of the present application.

[0049] Fig. 4 is a result diagram after S3 is completed in an application example of the lesion resection risk assessment method based on an endoscopic ultrasound image according to the present application.

[0050] Fig. 5 is a schematic diagram of the position of a sub-image in an ultrasound image in S4 in an application example of the lesion resection risk assessment method based on an endoscopic ultrasound image according to the present application.

[0051] Fig. 6 is a schematic diagram of S5.1 in an application example of the lesion resection risk assessment method based on an endoscopic ultrasound image according to the present application.

[0052] Fig. 7 is a structural block diagram of the lesion resection risk assessment system based on an endoscopic ultrasound image according to the present application. DETAILED DESCRIPTION

[0053] In order to enable a more clear understanding of the above-mentioned objects, features and advantages of the present application, the present application will be further described below with reference to the drawings and specific embodiments.

[0054] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, however, the present application can also be implemented in other ways different from those described herein, and therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0055] It should be noted that the description involving "first", "second" and the like in the present application is only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the technical features indicated or the number of technical features indicated. Therefore, the features limited by "first", "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of various embodiments can be combined with each other, but it must be based on the realization of ordinary skilled in the art, when the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, nor within the scope of protection claimed in the present application.

[0056] The lesion resection risk assessment method based on an endoscopic ultrasound image is used to fill the gap in the prior art, and provides a lesion resection risk assessment method to provide reliable perforation risk assessment for clinical treatment, help doctors to develop safer surgical plans, and reduce surgical risk.

[0057] Referring to Fig. 1, the lesion resection risk assessment method based on an endoscopic ultrasound image comprises:

[0058] S1, using a target detection model to detect targets in the endoscopic ultrasound image to obtain first and second detection results;

[0059] S2, cropping the endoscopic ultrasound image according to the first detection result to obtain a first target image, and cropping the endoscopic ultrasound image according to the second detection result to obtain a second target image;

[0060] S3, segmenting a landmark region in the second target image by using a semantic segmentation model to obtain a third target image;

[0061] S4, detecting a scale line segment in the first target image and calculating a physical length corresponding to each pixel as a first physical landmark parameter;

[0062] S5, extracting a feature parameter of the endoscopic ultrasound image, the feature parameter of the endoscopic ultrasound image including a physical length feature parameter;

[0063] S6, evaluating a perforation probability according to the feature parameter of the endoscopic ultrasound image.

[0064] It should be understood that the above method does not limit the above steps, and the execution process can be different, for example, S3 and S4 can be performed synchronously, so the above method does not represent or imply that all steps must be executed in this order, and a person skilled in the art can change or change the execution order of the above steps on the basis of the present application, and some embodiments of the above method will be illustrated below.

[0065] Referring to FIG. 2, FIG. 2 is a schematic diagram of the specific process of the above method.

[0066] The specific process of the lesion resection risk evaluation method based on the endoscopic ultrasound image is as follows:

[0067] S1, using a pre-trained deep learning-based target detection model, the target detection model can use one of the target detection algorithms such as yolo, ssd, faster-rcnn, etc., the target detection model receives an input endoscopic ultrasound image, and automatically detects the target to obtain a first detection result and a second detection result.

[0068] S2, the first detection result is the coordinate information of the effective area of the endoscopic ultrasound image, and the endoscopic ultrasound image is cropped according to the first detection result to obtain a first target image, that is, the first target image is the image of the effective area in the endoscopic ultrasound image, and the second detection result is the tumor region coordinate under the endoscope. The image of the tumor and its surrounding area is obtained as the second target image according to the tumor region coordinate. Referring to FIG. 3, the large green box in FIG. 3 is the first target image, and the small second target image.

[0069] Please refer to Fig. 3 again, according to the preset inflation coefficient, the tumor coordinate area is processed by inflation to obtain a new inflation coordinate, so that the new inflation coordinate area contains the image information of the tumor and the surrounding mucosa level. The inflation coefficient is preset, and the inflation coefficient used in the application is 1 / 2 of the width and height of the target image. The upper and lower parts of the second detection result are respectively inflated by 1 / 2 of the height, and the left and right parts are respectively inflated by 1 / 2 of the width, so as to obtain the tumor coordinate area and the surrounding area. This area is used as a pre-obtained area to crop the endoscopic ultrasound image to obtain a second target image. The second target image is an image of the tumor and its surrounding area in the endoscopic ultrasound image, which can be seen in Fig. 4.

[0070] S3, using a semantic segmentation model to segment the landmark area in the second target image to obtain a third target image. The landmark area includes background (background area), submucosal tumor area, mucosa layer area, muscularis mucosa area, submucosal layer area, muscularis propria area and serosa layer area.

[0071] Specifically, the semantic segmentation model can be used to segment and distinguish the landmark area to obtain the third target image.

[0072] Here, the semantic segmentation network is a pre-trained model, and the semantic segmentation network is trained using a pre-constructed segmentation mask data set to obtain the semantic segmentation model. The semantic segmentation network model can use UNet, SegNet, Deeplab, FCN, etc. The segmentation target of the semantic segmentation model includes: background, submucosal tumor area, mucosa layer area, muscularis mucosa area, submucosal layer area, muscularis propria area and serosa layer area. It can be understood that the second target image will have several of the background, submucosal tumor area, mucosa layer area, muscularis mucosa area, submucosal layer area, muscularis propria area and serosa layer area, and the semantic segmentation network model can use the color annotation of each area set by the semantic segmentation network model. The area can be set as the background area, and the image after segmentation and different annotation is the third target image, which can be seen in Fig. 4. The different annotation is not limited to this way.

[0073] S4, using an image smoothing processing algorithm and an edge extraction image processing algorithm to process the first target image, then detecting the scale line segment in the processed first target image, determining the pixel position information of the scale line segment, calculating the pixel distance of adjacent scale line segments, and calculating the physical length corresponding to each pixel as the first physical landmark parameter according to the actual physical distance between adjacent scale line segments.

[0074] Referring to FIG. 5, the scale line segment is a scale line in the endoscopic ultrasound image. The first target image is a rectangular frame in the total interface of the endoscopic ultrasound image after cutting off the parameters of detection date, time, FR, DG, MI, TIS, etc., but the first target image includes the scale line segment in the endoscopic ultrasound image. In the field, the scale line segment is located in at least one of the upper and lower edges of the first target image as a scale line on the horizontal coordinate axis; the scale line segment is also located in at least one of the left and right edges as a scale line on the vertical coordinate axis. In FIG. 5, the scale line is located in the upper edge and the right edge of the first detection result of the endoscopic ultrasound image, and the red box area in FIG. 5 is a sub-image.

[0075] Generally, the endoscopic ultrasound image has long scale lines and short scale lines, that is, a short scale line is between two long scale lines, and a long scale line is between two short scale lines.

[0076] Specifically, S4 is: the first target image is cropped to obtain a sub-image with a scale line segment, the sub-image is processed using an image smoothing processing algorithm and an edge extraction image processing algorithm, a straight line segment in the processed sub-image is detected, the straight line segment obtained by the clustering algorithm processing is classified to determine the pixel position information of the straight line segment, the pixel distance of adjacent straight line segments in the same direction is calculated, and the physical length corresponding to each pixel is calculated as the first physical marker parameter according to the actual physical distance of the scale line interval. The straight line segment is usually a scale line segment.

[0077] S4 includes:

[0078] S4.1, image cropping: according to prior information, the first target image is cropped to obtain a first sub-image and a second sub-image. The prior information is the position information of the scale line in the first target image, which can be determined by human visual observation and analysis. For example, the prior information is that the top and bottom of the first target image each have a scale line of the horizontal coordinate axis, the right side has a scale line of the vertical coordinate axis, and each is within 50 pixels. Then the specific operation of S4.1 is: cropping 50 pixels from the top and bottom of the first target image to obtain the first sub-image, and cropping 50 pixels from the rightmost side to the left to obtain the second sub-image. It can be understood that the first sub-image and the second sub-image each have a scale line segment.

[0079] S4.2, the first sub-image and the second sub-image are respectively processed as follows to obtain a first processed sub-image and a second processed sub-image: the images are converted into gray scale images, a 5x5 Gaussian filter is applied to smooth the gray scale images, a Canny edge detection operator is used for edge detection processing, and a more detailed binary image is obtained.

[0080] S4.3, detecting straight line segments in the first target image of the first processed sub-image and the second processed sub-image, classifying the straight line segments to obtain straight line segment positions, calculating pixel distances of adjacent straight line segments based on the straight line segment positions, and calculating a physical length corresponding to each pixel as the first physical mark parameter according to an actual physical distance of the scale line spacing.

[0081] S4.31, using a Hough straight line transform to detect straight line segments in the edge image of the first processed sub-image and the second processed sub-image. To ensure that only straight line segments are detected, the present application sets a minimum line segment length (such as 10 pixels) and a maximum line segment gap (such as 5 pixels), and filters out line segments with horizontal, vertical, near-horizontal, and near-horizontal-vertical directions. Here, for the first processed sub-image, line segments with vertical and near-vertical directions are filtered out, and for the second processed sub-image, line segments with horizontal and near-horizontal directions are filtered out.

[0082] S4.32, cluster analysis: through line segment cluster analysis, line segments with similar positions and consistent directions are classified into a class, and the cluster result is the pixel position information of each straight line segment. Near-horizontal and horizontal line segments belong to consistent directions, and near-vertical and vertical line segments belong to consistent directions.

[0083] Here, the line segment cluster analysis is classified according to length and position. The straight line segments in the same class not only have the same length (which can include similar lengths), but also have almost the same horizontal (or vertical) coordinate at one end and almost the same horizontal (or vertical) coordinate at the other end.

[0084] It can be understood that at least two classes are classified, one class is short scale lines, and one class is long scale lines, and the horizontal coordinates or vertical coordinates of the position coordinates of all long scale lines are almost the same, and the horizontal coordinates or vertical coordinates of the position coordinates of all short scale lines are almost the same. It can be understood that the sub-image can have other straight lines, which will also form several classes according to the situation. It can be understood that the class of scale lines can have other straight line segments in addition to scale line segments.

[0085] Preferably, S4.32 further includes a step of removing part of the cluster result classes according to the position coordinates of the straight line segments in the cluster result, i.e. removing classes that are not scale lines, and the removal basis can be quantity, uniformity of spacing, etc.

[0086] S4.33, calculating the pixel distance of adjacent straight line segments:

[0087] For the first sub-image: according to the pixel position information of the vertical straight line segments obtained by clustering, adjacent straight line segments are selected, and the pixel distance D_pixels between the center points of the adjacent straight line segments is calculated. For the second sub-image: according to the pixel position information of the horizontal straight line segments obtained by clustering, adjacent straight line segments are selected, and the pixel distance D_pixels between the center points of the adjacent straight line segments is calculated.

[0088] To improve the accuracy, the pixel distance of the multiple adjacent straight line segments is calculated from left to right and / or from top to bottom, and the binary method only calculates the local area, and the average value D_avg_pixels is obtained after the calculation. Preferably, not only the average value is considered, but also the mode is considered, and the pixel distance of the adjacent straight line segments is comprehensively analyzed and determined.

[0089] In a preferred embodiment, only the pixel distance of the adjacent long straight line segments (long scale lines) is calculated in this step, and the pixel distance of the adjacent short straight line segments is not calculated.

[0090] S4.34, physical length calculation: using the known physical scale length (the actual physical distance of the scale line spacing, such as the actual distance between two horizontal long scale lines is 10 mm, which is generally set by human and obtained in advance), the physical length L_per_pixel corresponding to each pixel is calculated = 10 mm / D_avg_pixels. This value is used as the first physical marker parameter in subsequent measurement.

[0091] It can be understood that if the length and width of each pixel are equal, only the first sub-image or the second sub-image can be obtained, or only the straight line segment with the horizontal direction or the vertical direction can be selected.

[0092] S5, extracting feature parameters according to the third target image, the second target image, etc., including the physical length feature parameter;

[0093] S5.1, extracting the first feature parameter, the second feature parameter and the third feature parameter in the third target image, wherein the first feature parameter and the second feature parameter belong to the physical length feature parameter: by calculating the vertical distance from the mucosa layer region to the top of the submucosal tumor region and the distance from the mucosa layer region to the serosa layer region in the third target image, the first, second and third feature parameters are generated. The confirmation method of the first, second and third feature parameters is shown in FIG. 6:

[0094] First, rotation correction: using the principal component analysis (PCA) algorithm, the principal axis direction of the mucosa layer region and the submucosal tumor region in the third target image is calculated. According to the principal axis direction, the rotation angle θ is calculated, and then the third target image is rotated according to the rotation angle θ to obtain the pretreated third target image, i.e. the third target image after rotation correction;

[0095] Secondly, the determination of the vertex is carried out: on the pretreated third target image, from the upper boundary of the mucosa layer region, the intersection point with the horizontal line is recorded as P0, and the first intersection point of the submucosal tumor region is found by scanning down row by row, and recorded as P1. Similarly, the last intersection point of the serosa layer region is scanned and recorded as P2. The mucosa layer region and the submucosal tumor region in the third target image can be extracted before the above method is performed.

[0096] Finally, referring to FIG. 6, the vertical distance calculation is carried out: the vertical distance between P0 and P1 is calculated to obtain L1; the vertical distance between P1 and P2 is calculated to obtain L2. In combination with the first physical marker parameter, the pixel distances L1 and L2 are converted into physical distances to obtain the first feature parameter and the second feature parameter; and the ratio of L1 to L2 is calculated and recorded as the third feature parameter.

[0097] The first feature parameter is referred to as the physical distance of the mucosa layer region and the submucosal tumor region, the second feature parameter is referred to as the physical distance of the submucosal tumor region and the serosa layer region, the second feature parameter contains the physical distance of the serosa layer region, and the third feature parameter is referred to as the ratio of the first feature parameter to the second feature parameter.

[0098] S5.2, extracting the fourth feature parameter and the fifth feature parameter: the fourth feature parameter and the fifth feature parameter are calculated according to the physical length of the long axis and the short axis of the tumor contour. This process converts the pixel length into the actual physical length by using the physical marker parameter.

[0099] Specifically:

[0100] On the pretreated third target image, the minimum circumscribed rectangle algorithm is used to determine the long axis and the short axis of the submucosal tumor region. Long axis calculation: the diagonal length of the minimum circumscribed rectangle is calculated as the long axis of the tumor. Short axis calculation: the short axis length is measured in the direction perpendicular to the long axis.

[0101] Physical length conversion: the pixel length of the long axis and the short axis of the tumor calculated above is converted into the physical long axis length of the tumor and the physical short axis length of the tumor by using the first physical marker parameter obtained previously, to obtain the fourth feature parameter and the fifth feature parameter. The fourth feature parameter is the physical length of the long axis of the submucosal tumor region, and the fifth feature parameter is the physical length of the short axis of the submucosal tumor region. The fourth feature parameter and the fifth feature parameter are physical length feature parameters.

[0102] S5.3, extracting the sixth feature parameter and the seventh feature parameter: based on the labeled data set, a CNN model (convolutional neural network model such as ResNet) is constructed and trained, a prediction model is used to predict the perforation risk of the second target image, and a perforation confidence and a non-perforation confidence are generated to obtain the sixth feature parameter and the seventh feature parameter.

[0103] S5.4, obtaining an eighth characteristic parameter, a ninth characteristic parameter and a tenth characteristic parameter: obtaining patient information corresponding to the endoscopic ultrasound image, a patient tumor occurrence site (for example, one of the upper part of the body of the stomach, the middle part of the body of the stomach, the lower part of the body of the stomach, the fundus of the stomach, the antrum of the stomach, the angle of the stomach), patient age, gender, which can be digitized, for example, patient gender is defined as 0 or 1, the upper part of the body of the stomach, the middle part of the body of the stomach, the lower part of the body of the stomach, the fundus of the stomach, the antrum of the stomach, the angle of the stomach are defined as numbers 2 to 9 respectively, the (digitized) patient tumor occurrence site as the eighth characteristic parameter, the Arabic numerals of the patient age as the ninth characteristic parameter, and the (digitized) patient gender as the tenth characteristic parameter.

[0104] S6, according to the first to tenth characteristic parameters, evaluating the perforation risk of each case;

[0105] The perforation risk of each case: using a pre-trained evaluation decision module by inputting the above-mentioned characteristic parameters into a decision model (such as XGBoost, decision tree, random forest, SVM, etc.), outputting the perforation risk probability.

[0106] The lesion resection risk evaluation method based on endoscopic ultrasound images of the present application combines target detection, semantic segmentation and feature extraction technology, not only provides an evaluation method for resection perforation risk, fills the medical gap, provides reliable perforation risk evaluation for clinical practice, helps doctors to develop safer surgical plans and reduces surgical risk; moreover, it realizes accurate prediction of perforation risk during surgical resection. The present application further improves the accuracy and reliability of perforation risk evaluation by combining a variety of characteristic parameters for comprehensive analysis. The present application realizes accurate conversion from pixels to physical length by determining the first physical marker parameter, which provides a solid foundation for subsequent quantitative analysis. The present application improves the safety of gastrointestinal tumor surgery, reduces the probability of intraoperative complications by accurately evaluating the perforation risk. The present application realizes efficient processing and analysis of ultrasound images, has high automation degree and can provide objective and quantitative risk evaluation basis for doctors. The present application provides a new and systematic evaluation method, which makes up for the shortcomings of the prior art in perforation risk prediction.

[0107] Referring to FIG. 7, the present application provides a lesion resection risk evaluation system based on endoscopic ultrasound images, comprising

[0108] The target detection module 10 is used for target detection of the endoscopic ultrasound image by using a target detection model to obtain a first detection result and a second detection result;

[0109] The cropping module 20 is used for cropping the endoscopic ultrasound image according to the first detection result to obtain a first target image, and cropping the endoscopic ultrasound image according to the second detection result to obtain a second target image;

[0110] The semantic segmentation module 30 is configured to segment a landmark region in the second target image by using a semantic segmentation model to obtain a third target image.

[0111] The physical landmark calculation module 40 is configured to detect scale line segments in the first target image and calculate a physical length corresponding to each pixel as a first physical landmark parameter.

[0112] The feature extraction module 50 is configured to extract a feature parameter of the endoscopic ultrasound image, and the feature parameter includes the physical length feature parameter.

[0113] The evaluation and decision module 60 is configured to evaluate a perforation probability according to the feature parameter.

[0114] The first target image includes scale line segments in the endoscopic ultrasound image; and the physical landmark calculation module 40 is specifically configured to crop the first target image to obtain a sub-image with scale line segments, process the sub-image by using an image smoothing processing algorithm and an edge extraction image processing algorithm, then detect straight line segments in the processed sub-image, determine pixel position information of the straight line segments, calculate a pixel distance of adjacent straight line segments, and calculate a physical length corresponding to each pixel according to an actual physical distance of the scale line intervals.

[0115] Further, the physical landmark calculation module 40 includes:

[0116] The first cropping unit is configured to crop the first target image to obtain a first sub-image and a second sub-image, and the first sub-image has scale lines of a horizontal coordinate axis, and the second sub-image has scale lines of a vertical coordinate axis.

[0117] The conversion unit is configured to convert the first sub-image and the second sub-image into gray-scale images.

[0118] The smoothing processing unit is configured to perform smoothing processing on the gray-scale images.

[0119] The edge detection unit is configured to perform edge detection processing on the images processed by the smoothing processing unit to obtain binary images, i.e., a first processed sub-image and a second processed sub-image.

[0120] The straight line detection unit is configured to detect straight line segments in the edge images of the first processed sub-image and the second processed sub-image.

[0121] The cluster analysis module is configured to perform line segment cluster analysis on the straight line segments to obtain pixel position information of the straight line segments.

[0122] The first calculation unit is configured to calculate a pixel distance of adjacent straight line segments in the same cluster of the line segment cluster analysis.

[0123] The second computing unit is configured to calculate the physical length corresponding to each pixel by using the actual physical distance of the known scale line distance, i.e., to obtain the first physical mark parameter.

[0124] The semantic segmentation module 30 is specifically configured to segment and distinguish the mark area in the second target image.

[0125] The feature extraction module 50 comprises:

[0126] The first feature extraction unit is configured to extract the first feature parameter, the second feature parameter and the third feature, specifically, to calculate the principal axis direction of the mucosa layer area and the submucosal tumor area, to rotate the third target image according to the principal axis direction to obtain the rotated third target image, to determine the distance of the mucosa layer area and the submucosal tumor area along the vertical direction, to determine the distance of the submucosal tumor area and the bottom end of the serosa layer area along the vertical direction, to calculate the first feature parameter and the second feature parameter according to the first physical mark parameter, and to calculate the third feature parameter.

[0127] The second feature extraction unit is configured to extract the fourth feature parameter and the fifth feature parameter, specifically, to determine the long axis and the short axis of the submucosal tumor area by using the minimum circumscribed rectangle algorithm on the rotated third target image, and the short axis is perpendicular to the long axis, and to calculate the physical length corresponding to the long axis and the short axis as the fourth feature parameter and the fifth feature parameter according to the first physical mark parameter.

[0128] The third feature extraction unit is configured to extract the sixth feature parameter and the seventh feature parameter, specifically, to predict the perforation risk of the second target image by using the prediction model and to generate the perforation confidence and the non-perforation confidence.

[0129] The fourth feature extraction unit is configured to extract the eighth feature parameter, the ninth feature parameter and the tenth feature parameter, specifically, to obtain the tumor occurrence site of the patient, the age of the patient and the gender of the patient, to digitally process the tumor occurrence site of the patient and the gender of the patient, and to take the digital tumor occurrence site of the patient as the eighth feature parameter and the digital gender of the patient as the tenth feature parameter.

[0130] In the implementation, the lesion resection risk assessment system based on the ultrasonic endoscopic image can be implemented by referring to the lesion resection risk assessment system based on the ultrasonic endoscopic image lesion resection risk assessment method in any of the above embodiments, and the specific implementation steps will not be described herein.

[0131] The application provides a lesion resection risk assessment device based on an endoscopic ultrasound image, comprising an image acquisition device and the lesion resection risk assessment system based on the endoscopic ultrasound image, wherein the image acquisition device is connected to an endoscopic ultrasound detector and the lesion resection risk assessment system based on the endoscopic ultrasound image, and the image acquisition device is used to acquire an endoscopic ultrasound detection video and acquire an endoscopic ultrasound image in the endoscopic ultrasound detection video.

[0132] Further, the device further comprises a display used for displaying the evaluation result of the lesion resection risk assessment system based on the endoscopic ultrasound image.

[0133] The application further provides a storage medium storing a computer program, wherein the computer program is executed by a processor to implement the steps of the lesion resection risk assessment method based on the endoscopic ultrasound image.

[0134] The processor can be a CPU, and can also be other general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0135] The memory can comprise various types of storage units, such as system memory, read-only memory, and permanent storage device. In addition, the memory can comprise a combination of any computer readable storage media, and the memory can be a semiconductor memory chip, a magnetic disk, an optical disk.

[0136] The executable code stored on the memory can make the processor execute part or all of the above-mentioned method when the executable code is processed by the processor.

[0137] Those skilled in the art in the field to which the application pertains can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the above-mentioned functions. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of the functional units and modules are only for the convenience of mutual distinction, and are not used to limit the protection scope of the application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0138] It should be noted that the description of the various embodiments is not exhaustive and that the description of one embodiment can be supplemented by the description of another embodiment.

[0139] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of other systems which are currently developed or later developed. Practitioners skilled in the art will recognize appropriate modifications, permutations, and combinations of these known functions and components. It is therefore intended that the application be considered as including all such variations and modifications. For example, the principles of the application can be applied to other communication systems of various types, including cellular telephone systems, satellite communication systems, and other mobile communication systems.

[0140] While the preferred embodiments of the application have been described, additional variations and modifications can be employed, as will be appreciated by those of ordinary skill in the art, once armed with the foregoing disclosure. Therefore, the appended claims are intended to embrace all such additional variations and modifications as well.

[0141] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for evaluating a risk of lesion resection based on an endoscopic ultrasound image, the method comprising: The method comprises the following steps: ​ target detection is performed on the endoscopic ultrasound image by using a target detection model to obtain a first detection result and a second detection result; the endoscopic ultrasound image is cropped according to the first detection result to obtain a first target image, and the endoscopic ultrasound image is cropped according to the second detection result to obtain a second target image; a semantic segmentation model is used to segment a landmark region in the second target image to obtain a third target image; a scale line segment in the first target image is detected, and a physical length corresponding to each pixel is calculated according to the scale line segment; feature parameters of the endoscopic ultrasound image are extracted, and the feature parameters include a physical length feature parameter; a perforation probability is evaluated according to the feature parameters.

2. The method of claim 1, wherein the method is based on an endoscopic ultrasound image. The first detection result is coordinate information of an effective region of the endoscopic ultrasound image, and the first target image is an image of the effective region in the endoscopic ultrasound image; the second detection result is a tumor region coordinate under the endoscopic ultrasound; and the second target image is an image of a tumor and a surrounding region in the endoscopic ultrasound image. 3.The method according to claim 1, wherein, The first target image includes a scale line segment in the endoscopic ultrasound image. The detection of the scale line segment in the first target image and the calculation of the physical length corresponding to each pixel according to the scale line segment specifically include the following steps: a sub-image with the scale line segment is obtained by cropping the first target image, the sub-image is processed by using an image smoothing processing algorithm and an edge extraction image processing algorithm, then a straight line segment in the processed sub-image is detected, pixel position information of the straight line segment is determined, a pixel distance between adjacent straight line segments is calculated, and a physical length corresponding to each pixel is calculated according to an actual physical distance of the scale line interval.

4. The method of claim 3, wherein the method is based on an endoscopic ultrasound image. The detection of the scale line segment in the first target image and the calculation of the physical length corresponding to each pixel according to the scale line segment specifically include the following steps: a first sub-image and a second sub-image are obtained by cropping the first target image, the first sub-image has a scale line of a horizontal coordinate axis, and the second sub-image has a scale line of a vertical coordinate axis; the first sub-image and the second sub-image are processed respectively to correspondingly obtain a first processed sub-image and a second processed sub-image: the images are converted into gray-scale images, the gray-scale images are smoothed, and then edge detection processing is performed to obtain binary images; straight line segments in edge images of the first processed sub-image and the second processed sub-image are detected; pixel position information of the straight line segments is obtained by performing line segment clustering analysis on the straight line segments; a pixel distance between adjacent straight line segments in the same class of the line segment clustering analysis is calculated; a physical length corresponding to each pixel is calculated by using an actual physical distance of a known scale line interval, that is, a first physical landmark parameter is obtained. 5.The method of claim 1, wherein, The segmentation of the landmark region in the second target image by using the semantic segmentation model specifically includes the following steps: the landmark region in the second target image is segmented and distinguished, and the landmark region includes a background, a submucosal tumor region, a mucosa layer region, a mucosa muscle layer region, a submucosa layer region, an intrinsic muscle layer region, and a serosa layer region. 6.The method according to claim 5, wherein, The feature parameters include part or all of a first feature parameter, a second feature parameter, a third feature parameter, a fourth feature parameter, a fifth feature parameter, a sixth feature parameter, a seventh feature parameter, an eighth feature parameter, a ninth feature parameter, and a tenth feature parameter. The first feature parameter is the physical distance between the mucosa layer region and the submucosal tumor region, the second feature parameter is the physical distance between the submucosal tumor region and the serosa layer region, the second feature parameter contains the physical distance of the serosa layer region, and the third feature parameter is the ratio of the first feature parameter to the second feature parameter; The fourth feature parameter is the physical length of the submucosal tumor region, and the fifth feature parameter is the physical length of the short axis of the submucosal tumor region; The sixth feature parameter is the perforation confidence, and the seventh feature parameter is the non-perforation confidence; The eighth feature parameter is the tumor occurrence site of the patient, the ninth feature parameter is the age of the patient, and the tenth feature parameter is the gender of the patient.

7. The method of claim 6, wherein the method is a method of evaluating the risk of resection of a lesion based on an endoscopic ultrasound image, the method comprising: obtaining an endoscopic ultrasound image of a lesion; and determining the risk of resection of the lesion based on the endoscopic ultrasound image. The extraction process of the first feature parameter, the second feature parameter and the third feature parameter is: The main axis direction of the mucosa layer region and the submucosal tumor region is calculated, and the third target image is rotated according to the main axis direction to obtain a rotated third target image; the distance between the mucosa layer region and the submucosal tumor region is determined along the vertical direction, the distance between the bottom end of the submucosal tumor region and the serosa layer region is determined along the vertical direction, the first feature parameter and the second feature parameter are calculated according to the first physical mark parameter, and the third feature parameter is calculated; The extraction process of the fourth feature parameter and the fifth feature parameter is: The minimum circumscribed rectangle algorithm is used on the rotated third target image to determine the long axis and the short axis of the submucosal tumor region, and the short axis is perpendicular to the long axis; the physical length of the long axis is calculated as the fourth feature parameter according to the first physical mark parameter, and the physical length of the short axis is calculated as the fifth feature parameter according to the first physical mark parameter; The extraction process of the sixth feature parameter and the seventh feature parameter is: predicting the perforation risk of the second target image by a prediction model and generating the perforation confidence and the non-perforation confidence; The extraction process of the eighth feature parameter, the ninth feature parameter and the tenth feature parameter is: obtaining the tumor occurrence site of the patient, the age of the patient and the gender of the patient, digitally processing the tumor occurrence site of the patient and the gender of the patient, the digital tumor occurrence site of the patient as the eighth feature parameter, the Arabic numerals of the age of the patient as the ninth feature parameter, and the digital gender of the patient as the tenth feature parameter.

8. A risk assessment system for lesion resection based on an endoscopic ultrasound image, characterized by, It comprises: A target detection module for performing target detection on the endoscopic ultrasound image by using a target detection model to obtain a first detection result and a second detection result; A cropping module for cropping the endoscopic ultrasound image according to the first detection result to obtain a first target image, and cropping the endoscopic ultrasound image according to the second detection result to obtain a second target image; A semantic segmentation module for segmenting the landmark region in the second target image by using a semantic segmentation model to obtain a third target image; A physical mark calculation module for detecting the scale line segment in the first target image and calculating the physical length corresponding to each pixel according to the scale line segment; A feature extraction module for extracting feature parameters of the endoscopic ultrasound image, wherein the feature parameters include physical length feature parameters; An evaluation and decision module for evaluating the perforation probability according to the feature parameters.

9. A lesion resection risk assessment device based on endoscopic ultrasound images, characterized in that, The system comprises an image obtaining device and the lesion resection risk assessment system based on endoscopic ultrasound images as claimed in claim 8, the image obtaining device is connected to an endoscopic ultrasound detector and the lesion resection risk assessment system based on endoscopic ultrasound images, and is used to obtain an endoscopic ultrasound detection video and an endoscopic ultrasound image in the endoscopic ultrasound detection video.

10. A storage medium storing a computer program, characterized by The computer program is executed by a processor to implement the steps of the lesion resection risk assessment method based on endoscopic ultrasound images as claimed in any one of claims 1-7.

Images