A deep learning-based biliopancreatic system ultrasonic endoscope navigation system and method

Abstract

This invention discloses a deep learning-based endoscopic ultrasound navigation system and method for the biliary and pancreatic system, relating to the field of smart healthcare. It includes: an acquisition and preprocessing module for real-time acquisition of sequential images of the biliary and pancreatic system during endoscopic ultrasound examination, and for performing bounding box removal, grayscale conversion, and pixel normalization on the acquired images; and a recognition module for deep feature extraction of the preprocessed images using a convolutional neural network architecture, completing the classification, recognition, and coordinate localization of 14 standard anatomical sites of the biliary and pancreatic system. This invention can perform precise preprocessing optimization of endoscopic ultrasound images of the biliary and pancreatic system, and based on multi-scale feature extraction, can complete the accurate classification, recognition, and actual anatomical coordinate localization of 14 standard anatomical sites. Combined with the full-sequence scanning logic and continuous navigation path constructed based on anatomical structure correlation, it is highly adaptable to the physiological structure of the biliary and pancreatic system and standard clinical scanning operations of endoscopic ultrasound.

Description

Technical Field

[0001] This invention relates to the field of smart medical technology, specifically to a deep learning-based endoscopic ultrasound navigation system and method for the biliary and pancreatic systems. Background Technology

[0002] The biliary and pancreatic system includes the gallbladder, bile ducts, and pancreas. Due to its deep anatomical location and the insidious nature of its lesions, diagnosis and treatment are quite challenging. Endoscopic ultrasound combines endoscopy with high-frequency ultrasound, allowing for scanning close to the biliary and pancreatic organs through the digestive tract. This provides clear visualization of the fine structures of the bile ducts and pancreatic parenchyma, accurately identifying lesions such as stones, tumors, and inflammation. It can also guide puncture biopsies and minimally invasive interventional treatments, making it a core technology for the diagnosis and treatment of biliary and pancreatic diseases.

[0003] The invention patent application with application number 202010416303.7 discloses a scanning and identification system and method for the normal anatomical structure of the biliary and pancreatic system. It aims to solve the problem that "although the existing technology can process the highlight spectrum with high efficiency, it is difficult to accurately identify ultrasound images because ultrasound images are usually single-channel images, which are more sensitive to spatial structural features. Moreover, the existing network model cannot focus on features with larger weights, so it cannot better reflect the spatial structural features of the image, resulting in low overall accuracy of the network model in identifying ultrasound images."

[0004] However, in the context of endoscopic ultrasound examination of the biliary and pancreatic systems, there is still considerable room for improvement in terms of accuracy and intelligence in the identification and localization of anatomical sites and the entire process navigation.

[0005] To this end, we propose a deep learning-based endoscopic ultrasound navigation system and method for the biliary and pancreatic systems. Summary of the Invention

[0006] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a deep learning-based endoscopic ultrasound navigation system and method for the biliary and pancreatic system, which can effectively solve the problems of the existing technology.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions; This invention discloses a deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems, comprising: The system comprises the following modules: an acquisition and preprocessing module for real-time acquisition of sequential images of the biliary and pancreatic system during endoscopic ultrasound, and for processing the acquired images by removing bounding boxes, converting to grayscale, and normalizing pixels; an identification module for deep feature extraction of the preprocessed images using a convolutional neural network architecture, classifying and locating 14 standard anatomical sites of the biliary and pancreatic system; a construction module for acquiring the identified standard anatomical sites, constructing a full-sequence scanning logic for the biliary and pancreatic system from the stomach to the duodenum, and generating continuous navigation path parameters between sites; a prompting module for receiving navigation path parameters and real-time site identification results, and outputting a visual site label and path guidance instruction for the current scanning position to the operator; a verification module for traversing and comparing the identified scanning sites during the examination based on a preset set of standard scanning sites of the biliary and pancreatic system to statistically analyze the scanned and unscanned site data; and an archiving module for receiving the entire process data of site identification, path navigation, and site verification, generating and outputting an endoscopic ultrasound navigation process data file. The acquisition and preprocessing module is interconnected with the identification module and the construction module via a wireless network. The construction module is interconnected with the prompting module and the verification module via a wireless network. The identification module, the construction module, the prompting module, and the verification module are interconnected with the archiving module via a wireless network.

[0008] Furthermore, the frame removal process performed by the acquisition and preprocessing module is to locate the edge contour of the effective imaging region based on the edge pixel gradient abrupt change characteristics and pixel value distribution characteristics of the effective imaging region of the ultrasound endoscope image, and then to crop the non-effective imaging region of the ultrasound endoscope image according to the edge contour coordinates obtained by the location, so as to complete the frame removal process. Grayscale processing involves converting the RGB three-channel pixel values ​​of color Doppler-type biliary and pancreatic system sequence images into grayscale images by performing nonlinear single-channel grayscale mapping based on the grayscale recognition requirements of biliary and pancreatic system tissue imaging, thereby completing the grayscale operation. Pixel normalization is an adaptive pixel normalization process that incorporates pixel neighborhood gradient features, and it follows the following rules: ; In the formula: The normalized pixel value of the pixel at coordinates (x, y) in the sequence image of the biliary and pancreatic system; This represents the original pixel value of the pixel at coordinates (x, y). The mean value of the original pixels in the neighborhood of the pixel at coordinates (x, y) is preset; Predetermine the standard deviation of pixel gradient within the neighborhood of the pixel at coordinates (x, y); This is the preset pixel reference value.

[0009] Furthermore, the convolutional neural network architecture in the recognition module is a multi-scale attention convolutional neural network architecture, which performs deep feature extraction on the preprocessed image as a fusion extraction of multi-scale convolutional feature maps, and completes the coordinate positioning of 14 standard anatomical sites of the biliary and pancreatic system as a quantization conversion from pixel coordinates to actual anatomical coordinates of the biliary and pancreatic system. The fusion of the multi-scale convolutional feature maps follows the following rules: ; The transformation from pixel coordinates to actual anatomical coordinates follows: ; In the formula: This is the fused feature map of the l-th convolution in a multi-scale attention convolutional neural network. This is the original feature map of the l-th convolutional layer; Let be the attention weight matrix of the l-th convolutional layer; The Hadamard product of matrices; These are the original feature maps of the (l-1)th and (l+1)th convolutional layers, respectively. This is a pixel-wise linear mapping after concatenating the channel dimensions of the feature maps; Let be the actual anatomical coordinates of the i-th standard anatomical site; , where are the x and y coordinates of the pixel at the i-th standard anatomical site in the preprocessed image; These are the horizontal and vertical pixel scaling factors for endoscopic ultrasound imaging, respectively. These are the correction values ​​for the horizontal and vertical coordinate deviations of endoscopic ultrasound imaging, respectively.

[0010] Furthermore, after the construction module obtains the identified standard anatomical sites, it first sorts the 14 standard anatomical sites sequentially from the stomach region to the duodenum region based on the anatomical structure correlation of the biliary and pancreatic system, and then constructs the full sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region. The continuous navigation path parameters include path node curvature, path node spacing, path travel direction, and path offset threshold. Among them, the path nodes are the sorted standard anatomical sites, the path node curvature is obtained by spatial geometric calculation based on the actual anatomical coordinates of three adjacent path nodes, the path node spacing is obtained by normalization based on the spatial distance of the actual anatomical coordinates of two adjacent path nodes, the path travel direction is the anatomical spatial direction from the previous path node to the next path node, and the path offset threshold is the allowable path deviation value set based on the preset scanning accuracy requirements.

[0011] Furthermore, after receiving the navigation path parameters and the real-time site recognition results, the prompting module first calculates the spatial deviation value between the actual anatomical coordinates of the real-time recognized site and the actual anatomical coordinates of the corresponding path node in the navigation path. Then, based on the comparison result of the spatial deviation value and the path offset threshold, it outputs the path travel guidance instruction and simultaneously performs visual site labeling of the current scanning position based on the feature identifier of the anatomical site.

[0012] Furthermore, the visualized site annotation is a multi-level layer fusion annotation, including an anatomical site name annotation layer, an anatomical site coordinate annotation layer, and a site recognition confidence annotation layer. Each annotation layer is pixel-fused with the real-time ultrasound endoscopic image layer according to a preset layer transparency. The path guidance instructions include a straight-ahead instruction, a fine-tuning instruction, a turnaround instruction, and a positioning instruction. When the spatial deviation value is less than the path offset threshold, a straight-ahead instruction is output. When the spatial deviation value is greater than or equal to the path offset threshold and less than the maximum deviation value of a preset interval, a fine-tuning instruction is output. When the spatial deviation value is greater than or equal to the maximum deviation value of a preset interval, a turnaround instruction is output. When no corresponding path node is identified, a positioning instruction is output.

[0013] Furthermore, when the verification module performs a traversal comparison of the scan sites identified during the examination based on the complete set of standard scan sites preset in the biliary and pancreatic system, it first establishes a feature matching library for site traversal comparison, and then matches the feature information of the identified scan sites with the feature information of the standard scan sites in the feature matching library one by one. If the match is successful, it is determined to be a scanned site; if the match fails, it is determined to be an unscanned site. Simultaneously, the verification module prioritizes and displays the statistics of unscanned sites. The priority of unscanned sites is calculated using the following formula: ; In the formula: This represents the priority value for the i-th unscanned site; is the anatomical importance coefficient of the biliary and pancreatic system at the i-th unscanned site; This is the normalized value for the spatial reachability of the i-th unscanned site relative to the current scanned location; This represents the sequential deviation between the i-th unscanned site and the current scanned node. This is the preset priority baseline value.

[0014] Furthermore, the archiving module receives full-process data for site identification, path navigation, and site verification, including site identification data, navigation path data, site verification data, and operation interaction data. The archiving module integrates the entire process data in a three-dimensional structure according to timestamps as the vertical dimension, module type as the horizontal dimension, and anatomical sites as the depth dimension. Then, based on the characteristic sparsity of the data, it compresses and encodes the data to generate and output an ultrasound endoscopic examination navigation process data file that conforms to the preset medical data format.

[0015] On the other hand, a deep learning-based endoscopic ultrasound navigation method for the biliary and pancreatic systems includes: The system acquires real-time endoscopic ultrasound images of the biliary and pancreatic systems, sequentially performing frame removal, grayscale conversion, and adaptive pixel normalization based on pixel neighborhood gradient features. Deep image features are extracted, and 14 standard anatomical sites are classified and identified, with the site pixel coordinates quantized and converted into actual anatomical coordinates. Based on the correlation of anatomical structures within the biliary and pancreatic systems, the identified sites are sequentially ordered, constructing a full-sequence scanning logic from the stomach to the duodenum, and simultaneously generating continuous navigation path parameters. Spatial deviations between real-time identified sites and navigation path nodes are calculated, and path guidance instructions are output. Simultaneously, multi-level layer fusion is performed to visualize the sites at the current scanning position. Feature information matching and comparison are performed between the identified scanned sites and the complete set of standard scanned sites, statistically analyzing scanned and unscanned site data, and calculating the priority of unscanned sites. The entire navigation process data is collected and integrated into a three-dimensional structure according to timestamp, data type, and anatomical site. After compression and encoding, a navigation process data file conforming to a preset medical data format is generated and output.

[0016] Compared with the known prior art, the technical solution provided by this invention has the following beneficial effects: This invention can perform precise preprocessing and optimization of endoscopic ultrasound sequence images of the biliary and pancreatic system. Based on multi-scale feature extraction, it can complete the accurate classification and identification of 14 standard anatomical sites and the actual anatomical coordinate positioning. Combined with the full-sequence scanning logic and continuous navigation path constructed by the correlation of anatomical structures, it is highly adapted to the physiological structure of the biliary and pancreatic system and the standard clinical scanning operation of endoscopic ultrasound. The system provides intuitive and accurate real-time guidance for inspection operations through multi-dimensional visual site annotation and hierarchical path guidance instructions. It can accurately verify the scanned sites and prioritize the unscanned sites based on the complete set of standard scanned sites, effectively reducing the problem of missed scans during inspection. The entire examination process data is integrated and archived in a three-dimensional structure, ensuring the complete storage and standardized management of medical data. The real-time and efficient transmission of data at each stage is maintained through wireless data interaction, which significantly improves the accuracy, standardization and operational efficiency of endoscopic ultrasound examination of the biliary and pancreatic system. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic diagram of a deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems. Figure 2 This is a flowchart illustrating a deep learning-based endoscopic ultrasound navigation method for the biliary and pancreatic systems. Figure 3 This is a schematic diagram of the biliary and pancreatic system. Figure 4 Endoscopic ultrasound scan of the biliary and pancreatic system Figure 1 ; Figure 5 Endoscopic ultrasound scan of the biliary and pancreatic system Figure 2 ; Figure 6 Endoscopic ultrasound scan of the biliary and pancreatic system Figure 3 ; Figure 7 Endoscopic ultrasound scan of the biliary and pancreatic system Figure 4 ; The labels in the diagram represent: 1. Liver and hepatic vein; 2. Abdominal aorta; 3. Pancreatic body, splenic artery, and splenic vein; 4. Pancreatic body and left kidney; 5. Left adrenal gland; 6. Splenic hilum and spleen; 7. The confluence of the mesenteric vein, superior mesenteric vein, and splenic vein; 8. First hepatic portal vein; 9. Portal vein branches; 10. Gallbladder; 11. Portal vein and bile duct; 12. Pancreatic head; 13. The confluence of the mesenteric vein, splenic vein, and portal vein; 14. Duodenal papilla. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] The present invention will be further described below with reference to embodiments.

[0021] Example 1: This embodiment presents a deep learning-based endoscopic ultrasound navigation method for the biliary and pancreatic systems, such as... Figure 1 As shown, it includes: The acquisition and preprocessing module is used to acquire sequential images of the biliary and pancreatic systems in real time during endoscopic ultrasound examinations, and to perform frame removal, grayscale conversion, and pixel normalization on the acquired images. The frame removal process performed by the acquisition and preprocessing module is to locate the edge contour of the effective imaging area based on the edge pixel gradient abrupt change characteristics and pixel value distribution characteristics of the effective imaging area of ​​the ultrasound endoscope image, and then to crop the non-effective imaging area of ​​the ultrasound endoscope image according to the edge contour coordinates obtained by the location to complete the frame removal process. Grayscale processing involves converting the RGB three-channel pixel values ​​of color Doppler-type biliary and pancreatic system sequence images into grayscale images by performing nonlinear single-channel grayscale mapping based on the grayscale recognition requirements of biliary and pancreatic system tissue imaging, thereby completing the grayscale operation. Pixel normalization is an adaptive pixel normalization process that incorporates pixel neighborhood gradient features, and it follows the following rules: ; In the formula: The normalized pixel value of the pixel at coordinates (x, y) in the sequence image of the biliary and pancreatic system; This represents the original pixel value of the pixel at coordinates (x, y). The mean value of the original pixels in the neighborhood of the pixel at coordinates (x, y) is preset; Predetermine the standard deviation of pixel gradient within the neighborhood of the pixel at coordinates (x, y); This is a preset pixel baseline value used to avoid the denominator being zero during normalization calculations; This formula subtracts the mean value of the original pixels in its preset neighborhood from the original pixel value, and then normalizes it by combining the standard deviation of the pixel gradient in the neighborhood. At the same time, it adds a preset pixel reference value to avoid the calculation problem of the denominator being zero. Unlike the traditional pixel standardization method that only relies on the distribution of pixel values, it incorporates the gradient features of the pixel neighborhood into the standardization process, so that the processed pixel values ​​are more in line with the imaging characteristics of the biliary and pancreatic system tissues, thereby effectively improving the accuracy of subsequent image feature extraction. The recognition module is used to extract deep features from the preprocessed image through a convolutional neural network architecture, and to complete the classification, recognition and coordinate localization of 14 standard anatomical sites of the biliary and pancreatic system. The convolutional neural network architecture in the recognition module is a multi-scale attention convolutional neural network architecture. It performs deep feature extraction on the preprocessed image as a fusion extraction of multi-scale convolutional feature maps, and completes the coordinate localization of 14 standard anatomical sites of the biliary and pancreatic system as a quantitative conversion from pixel coordinates to actual anatomical coordinates of the biliary and pancreatic system. The fusion of multi-scale convolutional feature maps follows the principle of: ; The above formula performs a Hadamard product operation on the original convolutional feature map of the l-th layer of the network and the corresponding attention weight matrix. At the same time, it integrates the results of the original convolutional feature maps of the (l-1)th and l+1th layers after channel-dimensional concatenation and pixel-wise linear mapping. It abandons the limitations of single-layer and single-scale feature extraction, highlights key features through the attention mechanism, and combines the complementary fusion of features from upper and lower layers to achieve comprehensive extraction of multi-scale convolutional features, so as to more accurately capture the deep image features of different anatomical sites of the biliary and pancreatic system. The transformation from pixel coordinates to actual anatomical coordinates follows: ; The above formula matches the horizontal and vertical coordinates of the anatomical site pixels with the corresponding imaging scaling factors, and then superimposes the coordinate deviation correction value calibrated by the standard anatomical phantom to achieve the quantitative conversion from pixel coordinates to actual anatomical coordinates. It sets a dedicated scaling factor for the difference in magnification of the horizontal and vertical imaging of the endoscopic ultrasound, and obtains the deviation correction value by calculating the mean value through the actual measured data of the phantom. This solves the problem of mismatch between the image pixel position and the actual anatomical position of the biliary and pancreatic system, making the site positioning results more in line with the actual clinical scenario. In the formula: This is the fused feature map of the l-th convolution in a multi-scale attention convolutional neural network. This is the original feature map of the l-th convolutional layer; Let be the attention weight matrix of the l-th convolutional layer; The Hadamard product of matrices; These are the original feature maps of the (l-1)th and (l+1)th convolutional layers, respectively. This is a pixel-wise linear mapping after concatenating the channel dimensions of the feature maps; Let be the actual anatomical coordinates of the i-th standard anatomical site; , where are the x and y coordinates of the pixel at the i-th standard anatomical site in the preprocessed image; These are the horizontal and vertical pixel scaling factors for endoscopic ultrasound imaging, respectively. These are the correction values ​​for the horizontal and vertical coordinate deviations of endoscopic ultrasound imaging, respectively. in, The values ​​are all within (0,1]. The higher the magnification of the transverse imaging of the biliary and pancreatic system by endoscopic ultrasound, the better. The larger the value, the lower the value. The smaller the value, the higher the magnification of the longitudinal imaging of the biliary and pancreatic system by endoscopic ultrasound. The larger the value, the lower the value. The smaller the value; The actual x-coordinates and calibrations of each point obtained from the standard anatomical phantom of the biliary and pancreatic system obtained by endoscopic ultrasound scanning are known. The mean value of the transformation deviation of the horizontal coordinates of all sites is calculated based on the imaging pixel's horizontal coordinate and the resulting image pixel's horizontal coordinate. Similarly; The module is used to acquire the identified standard anatomical sites, construct the full-sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region, and generate continuous navigation path parameters between sites; After the construction module obtains the identified standard anatomical sites, it first sorts the 14 standard anatomical sites from the stomach region to the duodenum region based on the correlation of the anatomical structure of the biliary and pancreatic system, and then constructs the full sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region. The anatomical correlation is a comprehensive correlation of the spatial adjacency of the anatomical positions of the biliary and pancreatic system from the stomach region to the duodenum region, the continuous course of the biliary and pancreatic tissues, and the anterograde relationship of the clinical standard scanning of endoscopic ultrasound. The sequential order is a site order that combines this comprehensive correlation to form a sequence that is compatible with the physiological structure of the biliary and pancreatic system and the clinical scanning operation of endoscopic ultrasound. Continuous navigation path parameters include path node curvature, path node spacing, path travel direction, and path offset threshold. Among them, the path nodes are the sorted standard anatomical sites, the path node curvature is obtained by spatial geometric calculation based on the actual anatomical coordinates of three adjacent path nodes, the path node spacing is obtained by normalization based on the spatial distance of the actual anatomical coordinates of two adjacent path nodes, the path travel direction is the anatomical spatial direction from the previous path node to the next path node, and the path offset threshold is the path deviation allowable value set based on the preset scanning accuracy requirements. The prompting module is used to receive navigation path parameters and real-time site recognition results, and output visual site annotations of the current scanning position and path travel guidance instructions to the inspection operator; After receiving the navigation path parameters and real-time site recognition results, the prompting module first calculates the spatial deviation between the actual anatomical coordinates of the real-time recognized site and the actual anatomical coordinates of the corresponding path node in the navigation path. Then, based on the comparison between the spatial deviation and the path offset threshold, it outputs the path travel guidance instruction and simultaneously performs visual site labeling of the current scanning position based on the feature identifier of the anatomical site. The visualization site annotation is a multi-level layer fusion annotation, including an anatomical site name annotation layer, an anatomical site coordinate annotation layer, and a site recognition confidence annotation layer. Each annotation layer is pixel-fused with the real-time ultrasound endoscopic image layer according to the preset layer transparency. The display style of the site identification confidence labeling layer is adaptively adjusted based on the comparison between the site identification confidence output by the identification module and a preset confidence threshold. The adjustment dimensions of the display style include label color, label brightness, label border line type, and label flashing frequency. When the site identification confidence is greater than or equal to the preset confidence threshold, the label color is a preset high-confidence base color, the label brightness is a preset high brightness value, and the label border is a preset solid line type with no flashing effect. When the site identification confidence is within a preset low-confidence interval, the label color is a preset medium-confidence gradient color, the label brightness is a preset medium brightness value, and the label border is a preset dotted line type with a low-frequency flashing effect. When the site identification confidence is less than the lower limit of the preset low-confidence interval, the label color is a preset low-confidence warning color, the label brightness is a preset low brightness value, and the label border is a preset double-dotted line type with a high-frequency flashing effect. The path guidance instructions include a straight-ahead instruction, a fine-tuning instruction, a turnaround instruction, and a positioning instruction. When the spatial deviation value is less than the path offset threshold, a straight-ahead instruction is output. When the spatial deviation value is greater than or equal to the path offset threshold and less than the maximum deviation value of the preset interval, a fine-tuning instruction is output. When the spatial deviation value is greater than or equal to the maximum deviation value of the preset interval, a turnaround instruction is output. When no corresponding path node is identified, a positioning instruction is output. The verification module is used to traverse and compare the scan sites identified during the examination based on the full set of standard scan sites preset in the biliary and pancreatic system, so as to statistically analyze the data of scanned and unscanned sites. When the verification module performs a traversal comparison of the scan sites identified during the examination based on the complete set of standard scan sites preset in the biliary and pancreatic system, it first establishes a feature matching library for site traversal comparison, and then matches the feature information of the identified scan sites with the feature information of the standard scan sites in the feature matching library one by one. If the match is successful, it is determined to be a scanned site; if the match fails, it is determined to be an unscanned site. The feature information includes anatomical attribute feature information, deep image recognition feature information, and spatial positioning feature information. The anatomical attribute features include: a unique anatomical identifier code for each of the 14 standard anatomical sites (corresponding one-to-one); the anatomical structure type of the standard anatomical site (such as vascular, organ parenchyma, lumen, confluence, etc.); the anatomical region affiliation of the standard anatomical site within the biliary and pancreatic system (stomach region / duodenal region); and the anatomical association attributes of the standard anatomical site (i.e., the association identifiers of adjacent anatomical structures corresponding to that site). The deep feature information for image recognition includes: the deep convolutional feature vector of the site extracted by the multi-scale attention convolutional neural network of the recognition module; the pixel texture feature set (including edge features, gray-scale distribution features, and morphological features) of the site in the preprocessed ultrasound endoscopic image; the recognition confidence value of the site output by the recognition module (the recognition confidence value is the classification probability value of the corresponding standard anatomical site of the biliary and pancreatic system output by the recognition module after performing softmax classification operation on the fused feature map of the preprocessed image through the multi-scale attention convolutional neural network); and the pixel range parameter of the effective imaging area of ​​the site in the ultrasound endoscopic image.

[0022] Spatial localization features include: the actual anatomical coordinates of the site in the biliary and pancreatic system after pixel coordinate transformation; the pixel coordinates of the site in the endoscopic ultrasound image; the spatial relative positional parameters (including spatial distance and azimuth) between the site and adjacent standard anatomical sites; and the sequential encoding of the site in the full-sequence scanning logic of the biliary and pancreatic system. Simultaneously, the verification module prioritizes and displays the statistics of unscanned sites. The priority of unscanned sites is calculated using the following formula: ; In the formula: For the first The priority value of each unscanned site; the higher the priority value, the higher the scanning priority. For the first Anatomical importance coefficients of the biliary and pancreatic system at unscanned sites; For the first The spatial reachability normalization values ​​of each unscanned site relative to the current scanned location follow the following rules: ; In the formula For the first The actual anatomical coordinate space distance between each unscanned site and the current scanned position. This is the set of all currently unscanned sites. This represents the maximum distance from the current scanning location among all currently unscanned sites. The range of values ​​is The larger the value, the closer the unscanned site is to the current scanned location and the higher its spatial accessibility. For the first The scanning sequence deviation value between each unscanned site and the current scanned node is a non-negative integer. The smaller the value, the smaller the sequential difference between the site and the current scanned node in the whole sequence scanning logic. This is a preset priority baseline value used to avoid the denominator being zero when calculating priority. This formula multiplies the anatomical importance coefficient of unscanned sites by their spatial reachability normalized value as the numerator, and uses the sum of the scan sequence deviation value and the preset benchmark value as the denominator to quantify the scan priority of unscanned sites: the higher the anatomical importance ( The larger the size, the closer the spatial distance to the current scanning location. The larger the value, the smaller the sequential deviation of the full sequence scan. The smaller the unscanned site, the higher its priority value. The larger the value, the higher the system priority will be, prompting the operator to perform a follow-up scan first; among which... The reverse mapping with maximum distance normalization is used to give higher accessibility scores to sites that are closer to each other, which is in line with the clinical efficiency principle of prioritizing the treatment of proximal missed sites and reducing repeated large movements of the endoscope in endoscopic ultrasound examination. By keeping the denominator positive, ensure that the sequential deviation value is... Priority calculation can still be performed normally when the value is zero (i.e., the unscanned site is the next site that should be scanned). By comprehensively considering the three core clinical factors of anatomical importance, spatial accessibility, and scanning sequence through the above technical solution, the operator is guided to prioritize the completion of key, adjacent, and sequentially close missed sites according to the optimal clinical logic, which effectively improves the efficiency and completeness of the examination. in, The stronger the correlation between the location and the core anatomical and physiological functions of the biliary and pancreatic system, and the higher the necessity for clinical examination and diagnosis of diseases of the biliary and pancreatic system, the larger its value; conversely, the smaller its value. The archiving module is used to receive the entire process data of site identification, path navigation and site verification, and generate and output the data file of the navigation process of endoscopic ultrasound examination. The archiving module receives full-process data for site identification, path navigation, and site verification, including site identification data, navigation path data, site verification data, and operation interaction data. The site identification data includes the identification results, identification timestamps, and actual anatomical coordinates of each standard anatomical site; the navigation path data includes navigation path parameters, path travel guidance instructions, and instruction output timestamps; the site verification data includes the statistical results of scanned / unscanned sites, the priority ranking results of unscanned sites, and verification timestamps; and the operation interaction data includes the operator's response to the guidance instructions and the response timestamps. The archiving module integrates the entire process data in a three-dimensional structure according to timestamp as the vertical dimension, module type as the horizontal dimension, and anatomical site as the depth dimension. Then, based on the characteristic sparsity of the data, it compresses and encodes the data to generate and output a data file of the ultrasound endoscopic examination navigation process that conforms to the preset medical data format. The acquisition and preprocessing modules are interconnected with the identification and construction modules via a wireless network. The construction module is interconnected with the prompting and verification modules via a wireless network. The identification, construction, prompting, and verification modules are interconnected with the archiving module via a wireless network.

[0023] In this embodiment, the acquisition and preprocessing module acquires sequential images of the biliary and pancreatic system in real time during endoscopic ultrasound examination and performs frame removal, grayscale conversion, and pixel normalization on the acquired images. The recognition module uses a convolutional neural network architecture to extract deep features from the preprocessed images, completing the classification, recognition, and coordinate positioning of 14 standard anatomical sites of the biliary and pancreatic system. After the construction module, only the recognized standard anatomical sites are allowed to be acquired. The full sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region is constructed, and continuous navigation path parameters between sites are generated. Then, the prompting module receives the navigation path parameters and real-time site recognition results, outputs the visual site label of the current scanning position and the path travel guidance instruction to the examination operator, and the verification module traverses and compares the scanned sites identified during the examination based on the preset set of standard scan sites of the biliary and pancreatic system to count the scanned and unscanned site data. Finally, the archiving module receives the full process data of site recognition, path navigation, and site verification, and generates and outputs the endoscopic ultrasound examination navigation process data file.

[0024] The system described in the above embodiments can accurately process examination sequence images, ensure efficient identification and accurate spatial positioning of standard anatomical sites, construct standardized full-sequence scanning logic based on anatomical structures and generate continuous navigation paths, reduce operational deviations through visual annotation and real-time path guidance, and comprehensively verify scanning sites to avoid missed scans.

[0025] It should be noted that: The 14 standard anatomical sites of the biliary and pancreatic system are as follows Figure 3 As indicated by the standard number.

[0026] Application examples (see) Figure 3 ): Each module works collaboratively according to the workflow to complete the inspection navigation. The specific application process is as follows: The system's acquisition and preprocessing module first acquires sequential images of the biliary and pancreatic systems in real time during the examination. The endoscopic ultrasound images of the biliary and pancreatic systems are as follows: Figure 4-7 As shown, where Figure 4 Figure (a) shows the hepatic vein and liver; Figure (b) shows the abdominal aorta; Figure (c) shows the pancreatic body, splenic artery and splenic vein; and Figure (d) shows the pancreatic body and left kidney. Figure 5The left adrenal gland is shown in Figure (a), the splenic hilum and spleen are shown in Figure (b), the superior mesenteric vein, splenic vein and portal vein are shown in Figure (c), and the portal vein, common bile duct and liver are shown in Figure (d). Figure 6 Figure (a) shows the portal vein branch (right branch), Figure (b) shows the liver and gallbladder, Figure (c) shows the portal vein and bile duct, and Figure (d) shows the head of the pancreas; Figure 7 The portal vein, splenic vein and superior mesenteric vein are shown in Figure (a), and the duodenal papilla and head of the pancreas are shown in Figure (b).

[0027] First, edge contours are located based on the edge pixel gradient abrupt change features and pixel value distribution features of the effective imaging area of ​​the endoscopic ultrasound image, and non-effective imaging areas are cropped to complete the bounding box process. Then, nonlinear single-channel grayscale mapping is performed on the color Doppler type image sequence to convert it into a grayscale image. Finally, adaptive pixel normalization processing combined with pixel neighborhood gradient features is performed to obtain the normalized pixel value corresponding to each pixel. Subsequently, the recognition module uses a multi-scale attention convolutional neural network architecture to perform multi-scale convolutional feature map fusion on the preprocessed image to extract deep features, complete the classification and recognition of 14 standard anatomical sites such as the liver, hepatic veins, and abdominal aorta, and at the same time, the pixel coordinates of each point are quantized and converted into actual anatomical coordinates to obtain the actual anatomical coordinate results corresponding to each of the 14 sites. After acquiring the recognition results of 14 standard anatomical sites, the construction module, based on the comprehensive correlation of the spatial adjacency of the anatomical locations from the stomach to the duodenum in the biliary and pancreatic system, the continuous course of biliary and pancreatic tissues, and the sequence of standard clinical endoscopic ultrasound scans, sorts the 14 sites sequentially from the stomach to the duodenum, constructing a full-sequence scanning logic for the biliary and pancreatic system. Simultaneously, it generates continuous navigation path parameters, calculating the corresponding values ​​of curvature and spacing between each path node, and determining the path direction and preset path offset threshold. The prompting module, after receiving the navigation path parameters and real-time site recognition results, calculates the spatial deviation between the actual anatomical coordinates of the real-time recognized site and the corresponding node in the navigation path. After comparing this deviation with the path offset threshold, it outputs corresponding path guidance instructions. Simultaneously, it performs multi-level layer fusion visualization of the site at the current scan position, fusing the anatomical site name, coordinates, and recognition confidence level with the real-time endoscopic ultrasound image layer at preset transparency. Furthermore, it adaptively adjusts the display style of the confidence level annotation layer based on the site recognition confidence results; for example, if the confidence level of liver and hepatic vein sites reaches a preset confidence level... The threshold is set as follows: The pancreatic head site has a low confidence level and is marked with a preset high-confidence base color, high brightness, solid line, and no flickering pattern. The pancreatic head site has a low confidence level and is marked with a preset medium-confidence gradient color, medium brightness, dotted line, and low-frequency flickering pattern. The gallbladder papilla site has a low confidence level and is marked with a preset low-confidence warning color, low brightness, double-dotted line, and high-frequency flickering pattern. If the real-time spatial deviation value is less than the path offset threshold, a straight-line command is output. If the deviation value reaches the threshold but is less than the maximum deviation value, a fine-tuning command is output. If the deviation value exceeds the maximum deviation value, a reversal command is output. If no corresponding path node is identified, a positioning command is output. The verification module is based on a complete set of standard scanning sites preset in the biliary and pancreatic system. First, it establishes a feature matching library for site traversal and comparison. Then, it matches the anatomical attributes, deep image recognition features, and spatial positioning features of the identified sites during the examination process with the standard site information in the library one by one. If the match is successful, it is determined as a scanned site; if the match fails, it is an unscanned site. At the same time, it calculates the priority value for the unscanned sites, obtains the priority result corresponding to each unscanned site, and displays it statistically according to the value. The larger the priority value, the higher the scanning priority.After receiving the entire process data of site identification, path navigation, site verification and operation interaction in this examination, the archiving module performs three-dimensional structured integration according to timestamp as the vertical dimension, module type as the horizontal dimension, and anatomical site as the depth dimension. After compression and encoding, it generates and outputs an ultrasound endoscopic examination navigation process data file that conforms to the preset medical data format.

[0028] Example 2: At the implementation level, based on Example 1, this example refers to... Figure 2 A further detailed description of the deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems in Example 1 is provided below: A deep learning-based endoscopic ultrasound navigation method for the biliary and pancreatic systems includes: Real-time acquisition of endoscopic ultrasound images of the biliary and pancreatic system, followed by frame removal, grayscale conversion, and adaptive pixel normalization processing incorporating pixel neighborhood gradient features. Deep features of the image were extracted to classify and identify 14 standard anatomical sites, and the pixel coordinates of the sites were quantized and converted into actual anatomical coordinates. Based on the correlation of the anatomical structures of the biliary and pancreatic system, the identification sites are sequentially ordered, and a full-sequence scanning logic from the stomach region to the duodenum region is constructed to generate continuous navigation path parameters simultaneously. Calculate the spatial deviation between the real-time identified location and the navigation path nodes, output the path travel guidance instructions, and simultaneously perform multi-level layer fusion visualization of the location annotation at the current scan position; The identified scanned sites are matched and compared with the complete set of standard scanned sites to collect data on scanned and unscanned sites, and the priority of unscanned sites is calculated. The entire navigation process data is collected and integrated in a three-dimensional structure according to timestamp, data type, and anatomical location. After compression and encoding, a navigation process data file conforming to the preset medical data format is generated and output.

[0029] In summary, the above embodiments can perform precise preprocessing and optimization of endoscopic ultrasound sequences of the biliary and pancreatic system. Based on multi-scale feature extraction, it can accurately classify and identify 14 standard anatomical sites and locate their actual anatomical coordinates. Combined with the full-sequence scanning logic and continuous navigation path constructed based on the correlation of anatomical structures, it is highly adapted to the physiological structure of the biliary and pancreatic system and standard clinical scanning operations of endoscopic ultrasound. Through multi-dimensional visualized site annotation and hierarchical path guidance instructions, it provides intuitive and accurate real-time guidance for the examination operation. It can accurately verify the scanned sites and prioritize the unscanned sites based on the complete set of standard scanned sites, effectively reducing the problem of missed scans during the examination. It performs three-dimensional structured integration and standardized compression and archiving of the entire examination process data, ensuring the complete storage and standardized management of medical data. Through wireless data interaction, it maintains the real-time and efficient transmission of data at each stage, significantly improving the accuracy, standardization, and operational efficiency of endoscopic ultrasound examination of the biliary and pancreatic system.

[0030] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems, characterized in that, include: The acquisition and preprocessing module is used to acquire sequential images of the biliary and pancreatic systems in real time during endoscopic ultrasound examinations, and to perform frame removal, grayscale conversion, and pixel normalization on the acquired images. The recognition module is used to extract deep features from the preprocessed image through a convolutional neural network architecture to complete the classification, recognition, and coordinate localization of standard anatomical sites of the biliary and pancreatic system. The module is used to acquire the identified standard anatomical sites, construct the full-sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region, and generate continuous navigation path parameters between sites; After the construction module obtains the identified standard anatomical sites, it first sorts the 14 standard anatomical sites sequentially from the stomach region to the duodenum region based on the anatomical structure correlation of the biliary and pancreatic system, and then constructs the full sequence scanning logic of the biliary and pancreatic system from the stomach region to the duodenum region. The continuous navigation path parameters include path node curvature, path node spacing, path travel direction, and path offset threshold. Among them, the path nodes are the sorted standard anatomical sites, the path node curvature is obtained by spatial geometric calculation based on the actual anatomical coordinates of three adjacent path nodes, the path node spacing is obtained by normalization based on the spatial distance of the actual anatomical coordinates of two adjacent path nodes, the path travel direction is the anatomical spatial direction from the previous path node to the next path node, and the path offset threshold is the path deviation allowable value set based on the preset scanning accuracy requirements. The prompting module is used to receive navigation path parameters and real-time site recognition results, and output visual site annotations of the current scanning position and path travel guidance instructions to the inspection operator; The verification module is used to traverse and compare the scan sites identified during the examination based on the full set of standard scan sites preset in the biliary and pancreatic system, so as to statistically analyze the data of scanned and unscanned sites. The archiving module is used to receive data from the entire process of site identification, path navigation, and site verification, and to generate and output data files for the navigation process of endoscopic ultrasound examination.

2. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, The frame removal process performed by the acquisition and preprocessing module is to locate the edge contour of the effective imaging area based on the edge pixel gradient abrupt change characteristics and pixel value distribution characteristics of the effective imaging area of ​​the ultrasound endoscope image, and then to crop the non-effective imaging area of ​​the ultrasound endoscope image according to the edge contour coordinates obtained by the location to complete the frame removal process. Grayscale processing involves converting the RGB three-channel pixel values ​​of color Doppler-type biliary and pancreatic system sequence images into grayscale images by performing nonlinear single-channel grayscale mapping based on the grayscale recognition requirements of biliary and pancreatic system tissue imaging, thereby completing the grayscale operation. Pixel normalization is an adaptive pixel normalization process that incorporates pixel neighborhood gradient features, and it follows the following rules: ； In the formula: The normalized pixel value of the pixel at coordinates (x, y) in the sequence image of the biliary and pancreatic system; This represents the original pixel value of the pixel at coordinates (x, y). The mean value of the original pixels in the neighborhood of the pixel at coordinates (x, y) is preset; Predetermine the standard deviation of pixel gradient within the neighborhood of the pixel at coordinates (x, y); This is the preset pixel reference value.

3. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, The convolutional neural network architecture in the recognition module is a multi-scale attention convolutional neural network architecture, which performs deep feature extraction on the preprocessed image to extract multi-scale convolutional feature maps and completes the coordinate localization of 14 standard anatomical sites of the biliary and pancreatic system as a quantization conversion from pixel coordinates to actual anatomical coordinates of the biliary and pancreatic system. The fusion of the multi-scale convolutional feature maps follows the following rules: ； The transformation from pixel coordinates to actual anatomical coordinates follows: ； In the formula: This is the fused feature map of the l-th convolution in a multi-scale attention convolutional neural network. This is the original feature map of the l-th convolutional layer; Let be the attention weight matrix of the l-th convolutional layer; The Hadamard product of matrices; These are the original feature maps of the (l-1)th and (l+1)th convolutional layers, respectively. This is a pixel-wise linear mapping after concatenating the channel dimensions of the feature maps; Let be the actual anatomical coordinates of the i-th standard anatomical site; , where are the x and y coordinates of the pixel at the i-th standard anatomical site in the preprocessed image; These are the horizontal and vertical pixel scaling factors for endoscopic ultrasound imaging, respectively. These are the correction values ​​for the horizontal and vertical coordinate deviations of endoscopic ultrasound imaging, respectively.

4. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, After receiving the navigation path parameters and real-time site recognition results, the prompting module first calculates the spatial deviation between the actual anatomical coordinates of the real-time recognized site and the actual anatomical coordinates of the corresponding path node in the navigation path. Then, based on the comparison result of the spatial deviation value and the path offset threshold, it outputs the path travel guidance instruction and simultaneously performs visual site labeling of the current scanning position based on the feature identifier of the anatomical site.

5. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 4, characterized in that, The visualized site annotation is a multi-level layer fusion annotation, including an anatomical site name annotation layer, an anatomical site coordinate annotation layer, and a site recognition confidence annotation layer. Each annotation layer is pixel-fused with the real-time ultrasound endoscopic image layer according to the preset layer transparency. The path guidance instructions include a straight-ahead instruction, a fine-tuning instruction, a turnaround instruction, and a positioning instruction. When the spatial deviation value is less than the path offset threshold, a straight-ahead instruction is output. When the spatial deviation value is greater than or equal to the path offset threshold and less than the maximum deviation value of a preset interval, a fine-tuning instruction is output. When the spatial deviation value is greater than or equal to the maximum deviation value of a preset interval, a turnaround instruction is output. When no corresponding path node is identified, a positioning instruction is output.

6. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, When the verification module performs a traversal comparison of the scan sites identified during the examination based on the complete set of standard scan sites preset in the biliary and pancreatic system, it first establishes a feature matching library for site traversal comparison, and then matches the feature information of the identified scan sites with the feature information of the standard scan sites in the feature matching library one by one. If the match is successful, it is determined to be a scanned site; if the match fails, it is determined to be an unscanned site. Simultaneously, the verification module prioritizes and displays the statistics of unscanned sites. The priority of unscanned sites is calculated using the following formula: ； In the formula: This represents the priority value for the i-th unscanned site; is the anatomical importance coefficient of the biliary and pancreatic system at the i-th unscanned site; This is the normalized value for the spatial reachability of the i-th unscanned site relative to the current scanned location; This represents the sequential deviation between the i-th unscanned site and the current scanned node. This is the preset priority baseline value.

7. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, The archiving module receives full-process data for site identification, path navigation, and site verification, including site identification data, navigation path data, site verification data, and operation interaction data. The archiving module integrates the entire process data in a three-dimensional structure according to timestamps as the vertical dimension, module type as the horizontal dimension, and anatomical sites as the depth dimension. Then, based on the characteristic sparsity of the data, it compresses and encodes the data to generate and output an ultrasound endoscopic examination navigation process data file that conforms to the preset medical data format.

8. The deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic systems according to claim 1, characterized in that, The acquisition and preprocessing module is interconnected with the identification module and the construction module via a wireless network. The construction module is interconnected with the prompting module and the verification module via a wireless network. The identification module, the construction module, the prompting module, and the verification module are interconnected with the archiving module via a wireless network.

9. A deep learning-based endoscopic ultrasound navigation method for the biliary and pancreatic system, wherein the method is an implementation method of the deep learning-based endoscopic ultrasound navigation system for the biliary and pancreatic system as described in any one of claims 1-8, characterized in that, include: Real-time acquisition of endoscopic ultrasound images of the biliary and pancreatic system, followed by frame removal, grayscale conversion, and adaptive pixel normalization processing incorporating pixel neighborhood gradient features. Extract deep features from the image, classify and identify standard anatomical sites, and quantize the site pixel coordinates into actual anatomical coordinates; Based on the correlation of the anatomical structures of the biliary and pancreatic system, the identification sites are sequentially ordered, and a full-sequence scanning logic from the stomach region to the duodenum region is constructed to generate continuous navigation path parameters simultaneously. Calculate the spatial deviation between the real-time identified location and the navigation path nodes, output the path travel guidance instructions, and simultaneously perform multi-level layer fusion visualization of the location annotation at the current scan position; The identified scanned sites are matched and compared with the complete set of standard scanned sites to collect data on scanned and unscanned sites, and the priority of unscanned sites is calculated. The entire navigation process data is collected and integrated in a three-dimensional structure according to timestamp, data type, and anatomical location. After compression and encoding, a navigation process data file conforming to the preset medical data format is generated and output.

Images