Axial self-attention based endoscopic ultrasound pancreas site identification method and system

Abstract

The application provides a pancreas endoscopic ultrasound station identification method and system based on axial self-attention, wherein the method comprises: acquiring a guide line composed of standard stations to be scanned and a pancreas endoscopic ultrasound image of the standard stations; classifying the pancreas endoscopic ultrasound image; constructing a first standard station identification network based on a cavity axial self-attention; inputting training sample images and test sample images into the first standard station identification network in sequence to obtain image training classification results; calculating a target loss function according to the training classification results and real station labels, and updating parameters of the first standard station identification network to obtain a second standard station identification network; and scanning the standard stations according to the guide line and prompting a scanning blind area by using the second standard station identification network. The application can automatically identify the examination blind area in the pancreas endoscopic ultrasound scanning process by constructing an efficient and accurate station identification model, thereby avoiding missed examination and missed diagnosis of the pancreas endoscopic ultrasound examination.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing, and particularly relates to a pancreatic endoscopic ultrasound station identification method and system based on axial self-attention. BACKGROUND

[0002] Pancreatic ductal adenocarcinoma (PDAC) is a digestive system malignant tumor with occult clinical manifestations, rapid development and extremely poor prognosis. Due to the fact that the specific symptoms of pancreatic cancer in the early stage are not significant and there is a lack of effective screening and early judgment means, many patients are in the advanced stage at the time of diagnosis, thus missing the best treatment period. Therefore, an effective PDAC early judgment method is of great significance to PDAC patients and their families.

[0003] PDAC early screening of high-risk groups has become a research hotspot and has been proven to significantly improve prognosis. Linear array EUS has many advantages for PDAC screening, and therefore high-quality linear array EUS pancreatic scanning is considered an indispensable tool for detection, qualitative and differential diagnosis of pancreatic diseases including PDAC. On the one hand, it is challenging for trainees to master EUS operation. Independent performance of safe and high-quality pancreatic EUS examination requires operators to master normal anatomy of the pancreas and interpretation ability of lesions from the cognitive level. On the other hand, there are deficiencies in the current pancreatic EUS scanning: 1) there is a lack of unified judgment index for standard pancreatic EUS scanning in the industry, which is extremely dependent on the subjective experience and feelings of the examining doctor, which also leads to subjective differences in the understanding of pancreatic EUS scanning images by different doctors; 2) poor quality of ultrasound images makes it difficult to determine the recognition site. Standardized pancreatic EUS scanning uses the station method, that is, the key principle to achieve efficient and complete pancreatic EUS is to cover all stations. SUMMARY

[0004] The application provides a pancreatic endoscopic ultrasound station identification method and system based on axial self-attention.

[0005] In a first aspect, the application provides a pancreatic endoscopic ultrasound station identification method based on axial self-attention, which comprises the following steps:

[0006] obtaining a guide line composed of standard stations scanned and a pancreatic endoscopic ultrasound image of the standard stations;

[0007] classifying the pancreatic endoscopic ultrasound image to obtain training sample images and test sample images;

[0008] constructing a first standard station identification network based on a cavity axial self-attention;

[0009] Input the training sample images and the test sample images into the first standard station identification network in sequence to obtain image training classification results;

[0010] Calculate a target loss function according to the training classification results and real station labels, and update parameters of the first standard station identification network to obtain a second standard station identification network;

[0011] According to the second standard station identification network, scan the standard stations according to the guide line and prompt the scanning blind area.

[0012] Further, the guide line composed of the scanned standard stations and the pancreas endoscopic ultrasound image of the standard station include:

[0013] Based on the station scanning principle and combined with the anatomical structure characteristics of the pancreas, the pancreas endoscopic ultrasound image is divided into 11 categories, including the pancreas head-suprahepatic vein and superior artery, the pancreas head-portal vein convergence area and common bile duct, the pancreas head-common bile duct and main pancreatic duct, the pancreas head-descending aorta, the pancreas neck-portal vein convergence, the pancreas neck-main pancreatic duct, the pancreas body-abdominal aorta and celiac trunk, the pancreas body-splenic vein and splenic artery, the pancreas tail-left kidney and the pancreas tail-spleen and the pancreas head-bulbar.

[0014] Further, the classification of the pancreas endoscopic ultrasound image to obtain the training sample images and the test sample images includes:

[0015] The effective area of the pancreas endoscopic ultrasound image is intercepted, the information frame irrelevant to the identification task is removed, and the size of the pancreas endoscopic ultrasound image after interception is unified;

[0016] The pancreas endoscopic ultrasound image with a unified size is classified to obtain the training sample images and the test sample images.

[0017] Further, the first standard station identification network includes an initial layer, an L-layer cavity axial self-attention block, a global average pooling layer and a linear fully connected layer.

[0018] Further, the initial layer is used to extract the local bottom features of the pancreas endoscopic ultrasound image with a unified size and perform spatial down-sampling, including a convolution layer, a batch normalization layer, a ReLU activation function and a global maximum pooling layer.

[0019] Further, the L-layer cavity axial self-attention block obtains a plurality of query-key-value triplets according to linear transformation, processes according to different attention head branches, splices the outputs of each branch and adds the original input features through residual connection to obtain the final output.

[0020] Further, the different attention head branches are stacked by row and column direction self-attention with different hole rates.

[0021] Further, the row direction self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional row direction window with holes centered thereon; and the column direction self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional column direction window with holes centered thereon.

[0022] Further, the target loss function adopts cross-entropy loss between the output result of the first standard site identification network and the real site label.

[0023] In a second aspect, the present application provides a pancreatic endoscopic ultrasound site identification method based on axial self-attention, comprising:

[0024] An acquisition module is configured to acquire a guide line composed of scanned standard sites and a pancreatic endoscopic ultrasound image of the standard sites;

[0025] A classification module is configured to classify the pancreatic endoscopic ultrasound image to obtain training sample images and test sample images;

[0026] A construction module is configured to construct a first standard site identification network based on a cavity axial self-attention;

[0027] An input module is configured to input the training sample images and the test sample images into the first standard site identification network in sequence to obtain image training classification results;

[0028] A calculation module is configured to calculate a target loss function according to the training classification results and real site labels, and update parameters of the first standard site identification network to obtain a second standard site identification network;

[0029] A scanning module is configured to scan standard sites according to the guide line by using the second standard site identification network and prompt a scanning blind area.

[0030] The application provides a pancreas endoscopic ultrasound site recognition method and system based on axial self-attention, wherein the method comprises: acquiring a guide line composed of scanned standard sites and a pancreas endoscopic ultrasound image of the standard sites; classifying the pancreas endoscopic ultrasound image to obtain training sample images and test sample images; constructing a first standard site recognition network based on a cavity axial self-attention; inputting the training sample images and the test sample images into the first standard site recognition network in sequence to obtain image training classification results; calculating a target loss function according to the training classification results and real site labels, and updating parameters of the first standard site recognition network to obtain a second standard site recognition network; and scanning the standard sites according to the guide line and prompting a scanning blind area by using the second standard site recognition network.

[0031] The site recognition network provided by the application adopts a cavity axial self-attention as a main feature extraction and transformation module, reduces the complexity of attention to linearity through axial decomposition and a cavity window, and enables each layer of attention calculation of the network to capture multi-scale context relationships by using multiple cavity rates. The application can automatically recognize a blind area in a pancreas endoscopic ultrasound scanning process by constructing an efficient and accurate site recognition model, assist a doctor in observation, avoid missed examination and missed diagnosis of pancreas endoscopic ultrasound examination, and help quality monitoring of pancreas endoscopic ultrasound scanning. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 A flowchart of a pancreas endoscopic ultrasound site recognition method based on axial self-attention provided for an embodiment of the application;

[0034] Figure 2 A pancreas endoscopic ultrasound scanning standard site schematic diagram provided for an embodiment of the application;

[0035] Figure 3 A structure diagram of a second standard site recognition network provided for an embodiment of the application;

[0036] Figure 4 A structure schematic diagram of a pancreas endoscopic ultrasound site recognition system based on axial self-attention provided for an embodiment of the application. DETAILED DESCRIPTION

[0037] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0038] In an embodiment, as shown in Figure 1 The embodiment of the present application provides a pancreatic endoscopic ultrasound station identification method based on axial self-attention, which comprises the following steps:

[0039] Step 101: acquiring a guide line composed of standard stations scanned and a pancreatic endoscopic ultrasound image of the standard station.

[0040] Based on the station scanning principle and combined with the anatomical structure characteristics of the pancreas, the pancreatic endoscopic ultrasound image is divided into 11 categories. As shown in the pancreatic EUS scanning standard station diagram, Figure 2 The 11 stations are C1 (pancreatic head--superior mesenteric vein and superior artery), C2 (pancreatic head--portal vein confluence and common bile duct), C3 (pancreatic head--common bile duct and main pancreatic duct), C4 (pancreatic head--descending aorta), C5 (pancreatic neck--portal vein confluence), C6 (pancreatic neck--main pancreatic duct), C7 (pancreatic body--abdominal aorta and celiac trunk), C8 (pancreatic body--splenic vein and splenic artery), C9 (pancreatic tail--left kidney), C10 (pancreatic tail--spleen), and C11 (pancreatic head--ampulla).

[0041] Exemplarily, for the collected samples, 7499 pancreatic endoscopic ultrasound images are used as a training set for network training, 1273 pancreatic endoscopic ultrasound images are used as a validation set for network hyperparameter selection, and 2504 pancreatic endoscopic ultrasound images are used as a test set for network diagnostic performance evaluation. The specific operation of preprocessing these pancreatic endoscopic ultrasound images is: intercepting the effective region of the pancreatic endoscopic ultrasound image, removing the information frame irrelevant to the identification task, and uniformly sizing the intercepted pancreatic endoscopic ultrasound image to 216*252 pixels.

[0042] Step 102: classifying the pancreatic endoscopic ultrasound image to obtain training sample images and test sample images.

[0043] Step 103: constructing a first standard station identification network based on a cavity axial self-attention.

[0044] As shown in Figure 3As shown, the first standard site identification network is stacked by an initial layer, a 6-layer cavity axial self-attention block, a global average pooling layer and a linear fully connected layer. The initial layer is composed of a convolutional layer, a batch normalization layer, a ReLU activation function and a global maximum pooling layer, aiming to extract local bottom features of the input image, such as texture, edge, etc. At the same time, the initial layer can also reduce the computational overhead of the subsequent modules of the network through spatial downsampling. The cavity axial self-attention block obtains multiple sets of query-key-value triplets through linear transformation, then processes them through different attention head branches, finally splices the outputs of each branch and adds them to the original input features through residual connection to obtain the final output. Different attention head branches are stacked by row and column direction cavity self-attention with different cavity rates. In this way, the multi-scale relationship can be captured during each forward calculation of the self-attention block. The attention head with a large cavity rate obtains a global feature, and the head with a small cavity rate obtains a local feature. Among them, the row direction cavity self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional row direction window with a cavity centered on it; the column direction cavity self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional column direction window with a cavity centered on it.

[0045] Step 104, input the training sample image and the test sample image into the first standard site identification network in turn to obtain an image training classification result.

[0046] Step 105, calculate a target loss function according to the training classification result and a real site label, and update the parameters of the first standard site identification network to obtain a second standard site identification network.

[0047] The random gradient descent with momentum is used as an optimization algorithm to update the parameters of the first standard site identification network. Further, the cross-entropy loss between the output result of the first standard site identification network and the real site label is used as the target loss function. The initial learning rate of the learning algorithm is 0.05, the momentum is 0.9, and the weight decay rate is 10-4. The learning rate adopts cosine decay, and the batch size is 64. According to the current training round number, it is judged whether the preset total training round number 100 is reached. If the training termination condition, i.e., the training round number reaches the total round number, is reached, the training is ended and the trained site identification network is output; otherwise, the training sample image is continuously input for training.

[0048] Step 106, using the second standard site identification network, the guide line is used to scan the standard site and prompt the scanning blind area.

[0049] Verify the effectiveness of the second standard site identification network: first, compare the second standard site identification network with the performance of the existing convolutional neural network and other self-attention methods, and the experimental results are shown in Table 1. It can be seen that the second standard site identification network has better prediction effect on the site identification task.

[0050] Table 1 Site prediction results of different models

[0051]

[0052] Based on the same inventive concept, the embodiment of the application also provides a pancreatic endoscopic ultrasound site identification system based on axial self-attention. Since the principle of solving the problem of the system is similar to the aforementioned pancreatic endoscopic ultrasound site identification method based on axial self-attention, the implementation of the system can be referred to the implementation of the aforementioned pancreatic endoscopic ultrasound site identification method based on axial self-attention, and the repeated parts will not be described here.

[0053] In another embodiment, the pancreatic endoscopic ultrasound site identification system based on axial self-attention provided by the embodiment of the application comprises: Figure 4 as shown in the figure, comprising:

[0054] The acquisition module 10 is configured to acquire a guide line composed of scanned standard sites and a pancreatic endoscopic ultrasound image of the standard sites.

[0055] The classification module 20 is configured to classify the pancreatic endoscopic ultrasound image to obtain training sample images and test sample images.

[0056] The construction module 30 is configured to construct a first standard site identification network based on a cavity axial self-attention.

[0057] The input module 40 is configured to input the training sample images and the test sample images into the first standard site identification network in sequence to obtain image training classification results.

[0058] The calculation module 50 is configured to calculate a target loss function according to the training classification results and a real site label, and update parameters of the first standard site identification network to obtain a second standard site identification network.

[0059] The scanning module 60 is configured to scan the standard sites according to the guide line by using the second standard site identification network and prompt a scanning blind area.

[0060] The more specific working process of each module described above can be referred to the corresponding content disclosed in the foregoing embodiments, which will not be described here.

[0061] In another embodiment, the present application provides a computer device comprising a processor and a memory; wherein the processor implements the steps of the above-mentioned method for recognizing a site of pancreas endoscopic ultrasound based on axial self-attention when executing a computer program stored in the memory.

[0062] More specific processes of the above-mentioned method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0063] In another embodiment, the present application provides a computer-readable storage medium for storing a computer program; the computer program is executed by a processor to implement the steps of the above-mentioned method for recognizing a site of pancreas endoscopic ultrasound based on axial self-attention.

[0064] More specific processes of the above-mentioned method can refer to the corresponding content disclosed in the foregoing embodiments, which will not be repeated here.

[0065] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system, device and storage medium disclosed by the embodiments, since they correspond to the method disclosed by the embodiments, the description is relatively simple, and the relevant part can be referred to the method part.

[0066] Those skilled in the art can clearly understand that the technology in the embodiments of the present application can be realized by means of software and necessary general hardware platform. Based on this understanding, the technical solutions in the embodiments of the present application can be embodied in the form of software product, which can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in the various embodiments or some parts of the embodiments of the present application.

[0067] The present application is described in detail above in combination with specific implementation manners and exemplary examples, but these descriptions cannot be understood as limitations of the present application. Those skilled in the art understand that the technical solutions and their implementation manners of the present application can be variously replaced, modified or improved without deviating from the spirit and scope of the present application, which all fall within the scope of the present application. The protection scope of the present application is subject to the appended claims.

Claims

1. An axial self-attention based endoscopic ultrasound pancreas site identification method, characterized by, include: Acquire the guide line consisting of standard scanning sites and the pancreatic endoscopic ultrasound images of the standard sites; The pancreatic endoscopic ultrasound images are classified to obtain training sample images and test sample images; Construct a first standard site identification network based on cavity axial self-attention; The training sample images and test sample images are sequentially input into the first standard site recognition network to obtain image training classification results; The target loss function is calculated based on the training classification results and the real site labels, and the parameters of the first standard site identification network are updated to obtain the second standard site identification network. The second standard site identification network is used to scan standard sites according to the guide lines and indicate blind spots. The step of classifying the pancreatic endoscopic ultrasound images to obtain training sample images and test sample images includes: cropping the effective region of the pancreatic endoscopic ultrasound images, removing information borders that are irrelevant to the recognition task, and standardizing the size of the cropped pancreatic endoscopic ultrasound images; classifying the pancreatic endoscopic ultrasound images of the standard size to obtain training sample images and test sample images. The first standard site identification network includes: an initial layer, an L-layer hole axial self-attention block, a global average pooling layer, and a linear fully connected layer; The L-layer cavity axial self-attention block obtains multiple sets of query-key-value triples according to the linear transformation, and processes them according to different attention head branches. The outputs of each branch are concatenated and added to the original input features through residual connection to obtain the final output. The different attention head branches are formed by stacking row-direction void self-attention and column-direction void self-attention with different void ratios; The row-oriented void self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional row-oriented window with holes centered on it; the column-oriented void self-attention means that the receptive field of each query element in the self-attention calculation is a one-dimensional column-oriented window with holes centered on it.

2. The pancreatic endoscopic ultrasound site identification method according to claim 1, characterized in that, The acquisition of the guide line composed of standard scanning stations and the pancreatic endoscopic ultrasound images of the standard stations includes: Based on the principle of site-based scanning and combined with the anatomical features of the pancreas, pancreatic endoscopic ultrasound images are divided into 11 categories, including pancreatic head—superior mesenteric vein and superior mesenteric artery, pancreatic head—portal vein confluence and common bile duct, pancreatic head—common bile duct and main pancreatic duct, pancreatic head—descending aorta, pancreatic neck—portal vein confluence, pancreatic neck—main pancreatic duct, pancreatic body—abdominal aorta and celiac trunk, pancreatic body—splenic vein and splenic artery, pancreatic tail—left kidney and pancreatic tail—spleen and pancreatic head—ampullary region.

3. The pancreatic endoscopic ultrasound site identification method according to claim 1, characterized in that, The initial layer is used to extract local low-level features from pancreatic ultrasound endoscopic images of uniform size and to perform spatial downsampling. It includes a convolutional layer, a batch normalization layer, a ReLU activation function, and a global max pooling layer.

4. The pancreatic endoscopic ultrasound site identification method according to claim 1, characterized in that, The target loss function uses the cross-entropy loss between the output of the first standard site identification network and the real site labels.

5. A system for pancreatic ultrasound endoscopic site identification based on the axial self-attention method according to any one of claims 1 to 4, characterized in that, include: The acquisition module is used to acquire the guide line composed of standard stations for scanning and the pancreatic endoscopic ultrasound images of the standard stations; The classification module is used to classify the pancreatic endoscopic ultrasound images to obtain training sample images and test sample images; The building block is used to construct a first standard site identification network based on void axis self-attention; The input module is used to sequentially input the training sample images and test sample images into the first standard site recognition network to obtain image training classification results; The calculation module is used to calculate the target loss function based on the training classification results and the real site labels, and update the parameters of the first standard site identification network to obtain the second standard site identification network. The scanning module is used to scan standard sites using a second standard site identification network following the guide lines and to indicate blind spots.

Images