Intelligent biopsy region prompting method and system for digestive endoscopy

Abstract

The application provides a kind of intelligent biopsy area prompting method and system for digestive endoscopy, it is related to artificial intelligence auxiliary diagnosis and treatment field, including: obtaining real-time digestive endoscopy video stream;Through multi-scale visual neural network, multi-scale feature extraction is carried out frame by frame to video stream, candidate biopsy area detection is carried out based on multi-scale feature;Biological histology level multi-feature fusion analysis is carried out to candidate biopsy area, and the biopsy value of candidate biopsy area is evaluated;Based on biopsy value, generate prompt element, and real-time superimposed in original video stream;Wherein, the biological histology level multi-feature fusion analysis is from color, texture, blood vessel, morphology, semantic multi-feature of candidate biopsy area is constructed, and the confidence of multi-feature and candidate biopsy area is used to score biopsy value.The application can accurately prompt potential biopsy area in the process of examination, so as to assist doctors to improve sampling scientificity and diagnostic accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence assisted diagnosis and treatment, in particular to an intelligent biopsy region prompting method and system for digestive endoscopy. BACKGROUND

[0002] At present, digestive endoscopy is an important auxiliary means for gastrointestinal disease diagnosis. Doctors usually need to judge suspicious areas during the examination process to decide whether to take a biopsy. In this process, the following problems exist:

[0003] (1) Different doctors have large differences in judging suspicious areas, resulting in poor diagnostic consistency. Due to subjective differences in the recognition standards of doctors with different seniority and experience for early lesions, subtle inflammation or tumor changes, the biopsy decision for the same suspicious area may be very different. This inconsistency in judgment not only affects the early detection rate of diseases, but also brings great difficulties to the standardized management and evaluation of medical quality.

[0004] (2) The endoscopic video frame rate is high, and the field of view changes dynamically, making it difficult for doctors to balance overall observation and local attention. Standard endoscopy produces a large number of real-time video frames, and factors such as mirror movement, organ peristalsis and mucus interference cause the field of view to change continuously and rapidly. In this environment, doctors need to constantly switch their attention between overall scanning of the mucosa and local detailed observation, and are highly stressed, making it easy to miss some suspicious areas that appear briefly or are not conspicuous due to visual fatigue or transient obstruction.

[0005] In summary, the prior art cannot effectively solve the problems of inconsistent biopsy decisions and potential missed detection caused by strong subjectivity and strong dynamics in endoscopy. SUMMARY

[0006] The present application proposes an intelligent biopsy region prompting method and system for digestive endoscopy to solve the above problems, which can prompt potential biopsy regions in real time and accurately during the examination process, thereby assisting doctors to improve the scientific nature of sampling and the accuracy of diagnosis.

[0007] According to some embodiments, the present application adopts the following technical solutions:

[0008] An intelligent biopsy region prompting method for digestive endoscopy, comprising:

[0009] acquiring a real-time digestive endoscopy video stream;

[0010] extracting multi-scale features from the preprocessed video stream frame by frame through a multi-scale visual neural network, and detecting candidate biopsy regions based on the multi-scale features;

[0011] The candidate biopsy region is subjected to multi-feature fusion analysis at a biological histology level to evaluate biopsy value of the candidate biopsy region.

[0012] Based on the biopsy value, a prompt element is generated and superimposed in the original video stream in real time.

[0013] The multi-feature fusion analysis at the biological histology level is multi-feature construction of the candidate biopsy region from color, texture, blood vessels, morphology and semantics, and biopsy value scoring is performed by using the multi-feature and confidence of the candidate biopsy region.

[0014] According to some embodiments, the present application adopts the following technical solutions:

[0015] An intelligent biopsy region prompting system for digestive endoscopy includes:

[0016] A video acquisition module is configured to acquire a real-time digestive endoscopy video stream.

[0017] A candidate detection module is configured to perform multi-scale feature extraction on the preprocessed video stream frame by frame by using a multi-scale visual neural network, and detect a candidate biopsy region based on the multi-scale feature.

[0018] A value evaluation module is configured to perform multi-feature fusion analysis at a biological histology level on the candidate biopsy region to evaluate biopsy value of the candidate biopsy region.

[0019] A biopsy prompting module is configured to generate a prompt element based on the biopsy value and superimpose the prompt element in the original video stream in real time.

[0020] The multi-feature fusion analysis at the biological histology level is multi-feature construction of the candidate biopsy region from color, texture, blood vessels, morphology and semantics, and biopsy value scoring is performed by using the multi-feature and confidence of the candidate biopsy region.

[0021] According to some embodiments, the present application adopts the following technical solutions:

[0022] A computer program product includes a computer program, which, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy.

[0023] According to some embodiments, the present application adopts the following technical solutions:

[0024] A non-transitory computer readable storage medium is used to store computer instructions, which, when executed by a processor, implement the intelligent biopsy region prompting method for digestive endoscopy.

[0025] According to some embodiments, the present application adopts the technical solutions as follows:

[0026] An electronic device comprises a processor, a memory and a computer program; wherein the processor is connected with the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to execute the intelligent biopsy region prompting method for digestive endoscopy.

[0027] Compared with the prior art, the present application has the beneficial effects that:

[0028] The present application objectively and quantitatively scores the biopsy value of the candidate biopsy region through a multi-dimensional quantitative evaluation method based on color, texture, blood vessel density, morphological irregularity and semantic features; this mechanism converts the judgment process originally relying on subjective experience into a standardized data-driven decision, effectively reducing the bias caused by human factors and significantly improving the consistency of diagnosis between different operators, thereby providing a reliable tool for the standardization and quality control of medical operations.

[0029] Through the multi-scale feature extraction and fusion technology, the present application can simultaneously capture global structural information and tiny local lesion features; in combination with real-time processing capability, each video frame is analyzed quickly and completely without omission, and the high-value region is color-coded and continuously tracked and prompted; the doctor is assisted to lock the target in a complex dynamic environment, and the risk of lesion missed detection caused by visual fatigue or instantaneous negligence is significantly reduced.

[0030] Through the lightweight network design and efficient algorithm process, the present application ensures that the system can stably run in the real-time video stream of endoscopy; the prompt elements are directly superimposed on the original gastroscope video picture familiar to the doctor in the form of frame prompt, region prompt and label prompt, which is intuitive and does not block the field of vision, realizes the "what you see is what you get" auxiliary effect, seamlessly accesses the clinical workflow without changing the existing operation habit of the doctor, and has high usability and deployment value. BRIEF DESCRIPTION OF DRAWINGS

[0031] The drawings accompanying the specification of the present application serve to provide a further understanding of the present application, and the illustrative embodiments of the present application and the description thereof serve to explain the present application, and do not constitute an improper limitation on the present application.

[0032] Figure 1 The method flowchart of example 1.

[0033] Figure 2 The multi-scale visual neural network structure diagram of example 1. DETAILED DESCRIPTION

[0034] The present application will be further described below in combination with the drawings and examples.

[0035] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0036] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, steps, operations, elements, components, and / or groups thereof, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0037] Example 1

[0038] In an embodiment of the present application, an intelligent biopsy region prompting method for digestive endoscopy is provided, which can prompt potential biopsy regions in real time and accurately during the examination process, thereby assisting doctors to improve the sampling scientificity and diagnostic accuracy. The specific steps are as follows:

[0039] Step S1: Obtain a real-time digestive endoscopy video stream;

[0040] The digestive endoscopy video stream is collected and preprocessed in real time, including operations such as illumination equalization, color standardization and denoising, to provide high-quality input for subsequent analysis.

[0041] Specifically, a multi-scale Retinex algorithm is used to correct the brightness and color constancy of each frame of image, to ensure the stability of the image texture features; for denoising and edge enhancement, bilateral filtering is used to remove high-frequency noise while preserving the tissue boundary; and then Laplacian enhancement filtering is used to highlight the mucosal surface texture.

[0042] Step S2: Perform multi-scale feature extraction on the preprocessed video stream frame by frame through a multi-scale visual neural network, and detect candidate biopsy regions based on the multi-scale features;

[0043] As shown in FIG. 1, the multi-scale visual neural network includes a multi-scale feature extraction unit, a segmentation and classification joint detection unit, and a candidate biopsy region set calculation unit. Figure 2 The multi-scale feature extraction unit is a lightweight visual Transformer and convolution network fusion architecture, which captures mucosal structure features of different scales.

[0044]

[0045] ​The segmentation and classification combined detection unit is used for segmenting and classifying the candidate regions of the extracted multi-scale features through a double-branch network structure, and finally outputs a pixel-level candidate region probability graph and a predicted abnormal type probability value of each candidate region.

[0046] The candidate biopsy region set calculation unit processes the pixel-level candidate region probability graph and the predicted abnormal type probability value of each region to obtain a candidate biopsy region, an abnormal type and a confidence, specifically as follows:

[0047] 1. Multi-scale feature extraction unit

[0048] A lightweight visual Transformer and convolution network fusion architecture is adopted to capture mucosa structure features of different scales.

[0049] Input image That is, an image frame in a video stream, after shallow convolution and block embedding, four levels of features are extracted:

[0050]

[0051] Among them, is used to capture low-level information such as color and brightness; is used to capture local edges and fine textures; is used to extract middle-level semantics such as tissue morphology and blood vessel direction; is used to capture global spatial context and lesion relationship.

[0052] Four levels of features are extracted through shallow convolution and block embedding, specifically as follows:

[0053] Shallow feature extraction: use depth separable convolution to reduce parameter quantity, extract and ;

[0054] Middle-level feature encoding: introduce multi-head self-attention mechanism to model long-range dependencies and extract ;

[0055] High-level semantic fusion: strengthen spatial context through window attention mechanism and extract ;

[0056] After extracting four levels of features, feature pyramid fusion is performed, and feature pyramid network is used to upsample different levels of features to the same scale, and to fuse features from bottom to top, specifically as follows:

[0057] 1) Upsample the deep features to the same spatial resolution as ;

[0058] 2) Fuse the upsampled With After channel concatenation, the fused feature is obtained by 1x1 convolution dimension reduction , which can be expressed as:

[0059]

[0060] Where Up represents up-sampling, and Concat represents channel concatenation.

[0061] 3) The up-sampled shallow feature is concatenated with the shallow feature After channel concatenation, the fused feature is obtained by 1x1 convolution dimension reduction , which can be expressed as:

[0062]

[0063] 4) The up-sampled shallow feature is concatenated with the shallow feature After channel concatenation, the fused feature is obtained by 1x1 convolution dimension reduction , which can be expressed as:

[0064]

[0065] This design combines the local stability of convolution and the global modeling ability of Transformer, and can effectively adapt to multi-scale targets under gastroscope view (such as small erosion, blood vessel dilation, local wrinkle abnormalities, etc.).

[0066] 2. Segmentation and classification joint detection unit

[0067] Based on the fused feature , the segmentation and classification of candidate regions are realized through a double-branch network structure.

[0068] The double-branch network adopts a shared backbone plus double-output head structure, including a segmentation branch and a classification branch. The segmentation branch generates a pixel-level candidate region probability map ; the classification branch outputs the predicted abnormal type probability value of each region .

[0069] Specifically, the segmentation branch up-samples the fused feature and performs convolution operation to output a pixel-level candidate region probability map consistent with the input size:

[0070]

[0071] Where is the Sigmoid function.

[0072] The classification branch first up-samples the fused feature ​The global semantic features are extracted by global average pooling (GAP) on the basis:

[0073]

[0074] Then, the prediction probability of each class is calculated through a fully connected layer:

[0075]

[0076] wherein, represents the probability value of different abnormal types (such as mucosal erosion, erythema, etc.) in the image, which is referred to as abnormal type, is the learned weight matrix and bias term parameter.

[0077] 3. Candidate biopsy region set calculation unit

[0078] After the above two branches, the generation step of the final candidate biopsy region is:

[0079] 1) For the pixel-level candidate region probability map wherein, is the probability that the pixel point in the image belongs to the candidate region, and the segmentation threshold is set to When , it is determined that the pixel belongs to the candidate region, thereby obtaining a binary mask, and further performing connected component analysis on the binary mask to obtain a connected region set , is the number of connected regions. 2) Calculate the average confidence of each region:

[0080]

[0081] The final confidence of each region is defined as:

[0082] wherein,

[0083] is a weighting coefficient (preferably 0.6), is the abnormal type confidence, and the calculation formula is:

[0084]

[0085] 3) If (for example, 0.6), the region is determined as a candidate biopsy region.

[0086] Output the candidate region set: wherein, is the candidate biopsy region,​ anomaly type, confidence.

[0087] The multi-scale visual neural network is trained by using an endoscope video frame data set labeled by medical experts. Since the candidate biopsy region set calculation unit is only numerical calculation and judgment, the multi-scale feature extraction unit and the segmentation and classification joint detection unit are mainly trained. The labeling content includes an abnormal region represented by a mask anomaly type (e.g., erosion, ulcer, polyp, etc.), and the data set is divided into a training set, a validation set, and a test set according to 8:1:1.

[0088] The loss function includes a classification loss term and a segmentation loss term. The formula of the classification loss term is:

[0089]

[0090] wherein, , are the predicted anomaly type and the labeled anomaly type, respectively.

[0091] The formula of the segmentation loss term is:

[0092]

[0093] wherein, , are the predicted abnormal region and the labeled abnormal region, respectively.

[0094] Step S3: performing multi-feature fusion analysis at a biological histology level on the candidate biopsy region to evaluate the biopsy value of the candidate biopsy region;

[0095] The multi-feature fusion analysis at the biological histology level is to construct multi-features of the candidate biopsy region from color, texture, blood vessels, morphology, and semantics, to use the multi-features and the confidence of the candidate biopsy region to score the biopsy value, and to divide the biopsy level (A / B / C level) according to the score interval.

[0096] The biopsy value score is calculated by extracting multi-features such as color, texture, blood vessels, morphology, and semantics for each candidate region. Specifically,

[0097] 1. Input and output definition

[0098] The input comes from the candidate biopsy region set of step S2

[0099] The output is the biopsy score of each region and the biopsy level .

[0100] 2. Multi-feature extraction unit

[0101] To realize scientific evaluation, multi-image features are extracted in each candidate region obtained in the input image , specifically:

[0102] (1) Color feature

[0103] Reflecting the change of mucosa color tone, bleeding tendency and surface redness, the calculation steps are:

[0104] 1) Convert the RGB image to HSV space;

[0105] 2) Calculate the hue deviation , saturation deviation and brightness deviation respectively, which can be expressed by the formula:

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115] wherein, , , denote the hue, saturation and brightness values extracted inside the candidate region, , and denote the hue, saturation and brightness values of normal gastric mucosa, which are calculated by taking the average of K points randomly sampled in the normal region outside the candidate region; H(x, y), S(x, y) and V(x, y) respectively denote the hue, saturation and brightness values of pixel point (x, y).

[0116] 3) After mean and standard deviation normalization, define:​

[0117]

[0118] (2) Texture feature

[0119] Reflecting the surface flatness and the uniformity of gland structure, the calculation steps are:

[0120] For the candidate region of the input image , the texture entropy , contrast and energy are calculated by using the gray level co-occurrence matrix, and the texture abnormality of the region is defined according to the above three features:

[0121]

[0122] wherein, are the weights of entropy, contrast and energy respectively, and the weight values are set by expert experience.

[0123] (3) Vessel feature

[0124] For the candidate region of the input image , the vessel feature in the region is evaluated from the vessel density ratio and the direction dispersion, and the calculation steps are: 1) For the candidate region of the input image

[0125] , the Frangi filter is used to enhance the vessel structure, and the binary image of the vessel region is extracted ; 2) Based on the binary image of the vessel region , the vessel density ratio is calculated:

[0126]

[0127]

[0128] wherein, represents the area of the vessel region in the candidate region, is the area of the candidate region.

[0129] 3) The vessel direction dispersion is calculated;

[0130] Based on the binary image of the vessel region , the local vessel direction angle is calculated at each vessel pixel by using the structure tensor method, and the average value of all direction angles in the region is counted ​​​; the vessel orientation dispersion is defined as the standard deviation of the set of orientation angles, calculated as:

[0131]

[0132] where N is the total number of orientation angles.

[0133] 4) Comprehensive index:

[0134]

[0135] where, are the weights of the vessel density ratio and the vessel orientation dispersion, respectively, and the weight values are set by expert experience.

[0136] (4) Morphological features Reflect the edge complexity and geometric morphology of the region, including edge curvature variation rate and region irregularity.

[0137] 1) Calculate the edge curvature variation rate ;

[0138] First, based on the mask of the candidate biopsy region , the region edge point set is extracted , where, is the number of point set elements. For each edge point, the first and second derivatives and are calculated by difference method, and the local curvature is calculated according to the definition of differential geometry:

[0139]

[0140] Then calculate the average value of all curvatures , and define the edge curvature variation rate as the average value of the absolute deviation of each point curvature and the average curvature, which is expressed as:

[0141]

[0142] 2) Calculate the irregularity , where, is the perimeter of the region edge, is the area of the region;

[0143] 3) Comprehensive definition of morphological features:

[0144]

[0145] where, are the weights of the edge curvature variation rate and the irregularity, respectively, and the weight values are set by expert experience.

[0146] (5) Semantic features

[0147] Reflecting the high-dimensional semantic understanding of the deep features of the network to the lesion area, the calculation steps are:

[0148] 1) Extract the high-level semantic vector corresponding to the region from the fusion feature map , specifically:

[0149] Average pooling is performed on the features in the region :

[0150]

[0151] 2) Use the weight matrix learned by the classification branch and the class probability to calculate the channel importance weight , and the calculation formula is:

[0152]

[0153] 3) Comprehensive definition:

[0154]

[0155] 3. Biopsy value scoring function

[0156] Based on the above features, the biopsy value scoring function is defined as:

[0157]

[0158] wherein, is the feature weight parameter, is the bias parameter, and the weight and bias parameters are determined by the labeled samples through a logistic regression model.

[0159] This embodiment uses a logistic regression model to determine the weight parameter and the bias parameter with expert-labeled training sample set, specifically:

[0160] (1) Training sample preparation

[0161] 1) Select a gastroscope image dataset confirmed by pathology.

[0162] 2) Each sample corresponds to one or more labeled biopsy regions , and has a manually labeled biopsy label , indicates that biopsy is needed (positive region), indicates that biopsy is not needed (negative or normal region).

[0163] 3) For each labeled region Color features, texture features, vessel features, shape features and semantic features are extracted to form a five-dimensional feature vector.

[0164]

[0165] To eliminate the differences in dimensions and value ranges of different features, zero-mean normalization is performed on the five types of features of all samples.

[0166] (2) Logistic regression model

[0167] To determine the contribution of each type of feature to the biopsy value, the logistic regression model is used for parameter estimation in this embodiment, and the model is defined as follows:

[0168]

[0169] The loss function adopts the negative log-likelihood form of the logistic regression, which is defined as:

[0170]

[0171] wherein, is the biopsy probability predicted by the model.

[0172] The parameters are solved by gradient descent or quasi-Newton optimization algorithm:

[0173]

[0174] After training, the coefficient vector is obtained, which represents the contribution of each feature to . Positive coefficients represent positive correlation between the feature and the biopsy demand, and negative coefficients represent negative correlation.

[0175] (3) Validation and normalization of the logistic regression model

[0176] To ensure the stability and generalization of the model:

[0177] 1) Five-fold cross validation is used to calculate the consistency of the model on the training set and the validation set.

[0178] 2) The feature weight obtained by the logistic regression is normalized, and the normalization method is:

[0179]

[0180] Similarly, can be obtained, and it satisfies:

[0181]

[0182] The weight of the normalized candidate is substituted into the biopsy value scoring function to obtain a normalized biopsy score:

[0183]

[0184] To enhance stability, the biopsy score and the candidate confidence are weighted and fused to obtain a final biopsy score:

[0185]

[0186] wherein, is the fusion weight, which is set to 0.6 in this embodiment.

[0187] The final biopsy value result is as follows:

[0188]

[0189] wherein, is the candidate biopsy region; is the biopsy grade, which is divided according to the score.

[0190] Step S4: Based on the biopsy value, a prompt element is generated and superimposed in the original video stream in real time;

[0191] During the gastroscopy process, the prompt elements (including the frame prompt, the region prompt, and the label information) are generated according to the candidate biopsy region and its biopsy score and grade output by step S3, and these prompt elements are superimposed into the original gastroscopy video stream in a low-latency manner to realize real-time prompting, specifically as follows:

[0192] 1. Prompt element generation

[0193] For each candidate biopsy region , the corresponding prompt element is generated according to its biopsy score and grade, including:

[0194] Frame prompt: used to mark the spatial position of the candidate biopsy region, acting on the outer contour of the region. Different grades adopt different frame colors and frame styles: grade A adopts a red solid line, grade B adopts a yellow dashed line, and grade C does not display the frame, thereby generating a separate frame layer .

[0195] Region prompt: used to visually enhance the inside of the candidate biopsy region, highlighting or weakening the abnormal features in the region according to the risk degree, generating a region layer including a binary mask , and according to the final biopsy score , dynamically adjust the transparency or color intensity of the region layer, and realize the expression of the saliency of the abnormal features in the region: A-level superimposed region enhancement, B-level only mild prompt, C-level superimposed semi-transparent weakening layer, and reduced visual saliency of the region.

[0196] Label information: display the biopsy score and level (such as "0.87 / A-level"), and generate a separate label layer .

[0197] 2. Dynamically stable display

[0198] To avoid the prompt box flickering or drifting in the gastroscope video sequence, a time smoothing strategy is adopted for the position of the prompt element:

[0199]

[0200] wherein, is the detection result of the new frame, is the prompt position of the last frame.

[0201] 3. Visualization superposition

[0202] An overlay layer structure is adopted to render the generated prompt information to the gastroscope video in real time, specifically:

[0203] The original gastroscope video frame layer is fused with the region layer through weighted transparency superposition, and is superimposed with the border layer and the label layer to obtain the final display frame , which can be expressed by the formula as:

[0204]

[0205] wherein, is the transparency weight parameter, which can be dynamically adjusted according to the biopsy score , and can be expressed by the formula as:

[0206]

[0207] The higher the biopsy score, the higher the transparency of the corresponding region and the brighter the color, so as to strengthen the visual prominence effect of the high-risk region.

[0208] The above generated prompt information is displayed on the region mask layer, and the output frame after final superposition retains the real texture of the original gastroscope image, and intuitively highlights the biopsy key region, which can realize stable and clear real-time prompting during the observation of the doctor.

[0209] Example 2

[0210] An embodiment of the present application provides an intelligent biopsy region prompting system for digestive endoscopy, comprising:

[0211] a video acquisition module configured to acquire a real-time digestive endoscopy video stream;

[0212] a candidate detection module configured to perform multi-scale feature extraction on the preprocessed video stream frame by frame through a multi-scale visual neural network, and perform candidate biopsy region detection based on the multi-scale features;

[0213] a value evaluation module configured to perform multi-feature fusion analysis of biological histology on the candidate biopsy region, and evaluate biopsy value of the candidate biopsy region;

[0214] a biopsy prompting module configured to generate a prompt element based on the biopsy value and superimpose the prompt element in the original video stream in real time;

[0215] The multi-feature fusion analysis of biological histology is multi-feature construction of the candidate biopsy region from color, texture, blood vessels, morphology and semantics, and biopsy value scoring is performed by using the multi-feature and confidence of the candidate biopsy region.

[0216] Embodiment 3

[0217] An embodiment of the present application provides a computer program product comprising a computer program, which, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy.

[0218] Embodiment 4

[0219] An embodiment of the present application provides a non-transitory computer readable storage medium for storing computer instructions, which, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy.

[0220] Embodiment 5

[0221] An embodiment of the present application provides an electronic device comprising a processor, a memory and a computer program; wherein the processor is connected with the memory, and the computer program is stored in the memory; when the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device performs the intelligent biopsy region prompting method for digestive endoscopy.

[0222] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts described above. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0223] These computer program instructions can also be loaded into a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks or in conjunction with the flowcharts described above. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks.

[0224] The above description is only specific embodiments of the present application, and is not intended to limit the protection scope of the present application. Any modifications or changes made by those skilled in the art based on the technical solutions of the present application without creative efforts shall fall within the protection scope of the present application.

Claims

1. A method for intelligent biopsy region prompting for digestive endoscopy, characterized in that, The method comprises the following steps: acquiring a real-time digestive endoscopy video stream; performing multi-scale feature extraction on the preprocessed video stream frame by frame through a multi-scale visual neural network, and detecting a candidate biopsy region based on the multi-scale features; wherein the multi-scale visual neural network comprises a multi-scale feature extraction unit, a segmentation and classification joint detection unit, and a candidate biopsy region set calculation unit; the multi-scale feature extraction unit adopts a lightweight visual Transformer and convolution network fusion architecture to capture mucosa structure features of different scales; the segmentation and classification joint detection unit performs segmentation and classification of the extracted multi-scale features through a double-branch network structure, and finally outputs a pixel-level candidate region probability map and a predicted abnormal type probability value of each candidate region; the candidate biopsy region set calculation unit processes the pixel-level candidate region probability map and the predicted abnormal type probability value of each region to obtain a candidate biopsy region, an abnormal type, and a confidence level; performing multi-feature fusion analysis of the candidate biopsy region at a biological histology level to evaluate the biopsy value of the candidate biopsy region; generating a prompt element based on the biopsy value and superimposing it in the original video stream in real time; wherein the multi-feature fusion analysis at the biological histology level is to construct multi-features of the candidate biopsy region from color, texture, blood vessels, morphology, and semantics, and to score the biopsy value using the multi-features and the confidence level of the candidate biopsy region; the semantic feature is a high-level semantic vector extracted from the multi-scale features, and an importance weight is calculated to reflect the high-dimensional semantic understanding of the network deep layer on the candidate biopsy region; the scoring of the biopsy value using the multi-features and the confidence level of the candidate biopsy region is represented by the following formula: wherein, , are the initial and final biopsy scores, respectively, are the color feature, texture feature, vessel feature, morphology feature, semantic feature, respectively, is the feature weight parameter, is the bias term parameter, is the confidence of the candidate biopsy region, is the weight parameter of the initial biopsy score.

2. The intelligent biopsy region prompting method for digestive endoscopy according to claim 1, wherein, the preprocessing includes frame extraction, denoising, illumination correction, and boundary enhancement.

3. The intelligent biopsy region prompting method for digestive endoscopy according to claim 1, wherein, The multi-features of the candidate biopsy region constructed from color, texture, blood vessels, morphology, and semantics are as follows: the color feature is a chroma deviation calculated to reflect mucosa tone changes, bleeding tendency, and surface redness; the texture feature is an entropy, contrast, and energy extracted using a gray level co-occurrence matrix to calculate a texture abnormality degree, which is used to reflect surface flatness and gland structure uniformity; the blood vessel feature is a blood vessel density ratio and a blood vessel direction dispersion degree calculated to evaluate the distribution, density, and twisting degree of blood vessels in the region; the morphology feature is an edge curvature change rate and irregularity calculated to reflect the edge complexity and geometric morphology of the region.

4. The intelligent biopsy region prompting method for digestive endoscopy according to claim 1, wherein, The generation of the prompt element includes setting a bounding box prompt, a region prompt, and label information.

5. A smart biopsy region prompting system for digestive endoscopy, characterized by, An intelligent biopsy region prompting method for digestive endoscopy according to any one of claims 1-4 comprises: a video acquisition module configured to acquire a real-time digestive endoscopy video stream; a candidate detection module configured to perform multi-scale feature extraction on the preprocessed video stream frame by frame through a multi-scale visual neural network, and detect a candidate biopsy region based on the multi-scale features; a value evaluation module configured to perform multi-feature fusion analysis of the candidate biopsy region at a biological histology level to evaluate the biopsy value of the candidate biopsy region; The biopsy prompt module is configured to generate a prompt element based on the biopsy value and superimpose the prompt element in the original video stream in real time. The multi-feature fusion analysis at the biological histology level is to construct multi-features of the candidate biopsy region from color, texture, blood vessels, morphology and semantics, and to score the biopsy value by using the multi-features and the confidence of the candidate biopsy region.

6. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy according to any one of claims 1-4.

7. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium is used to store computer instructions, and the computer instructions, when executed by a processor, implement the intelligent biopsy region prompting method for digestive endoscopy according to any one of claims 1-4.

8. An electronic device, comprising: The computer program, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy according to any one of claims 1-4. The computer program, when executed by a processor, implements the intelligent biopsy region prompting method for digestive endoscopy according to any one of claims 1-4.

Images