An oral cavity recognition method based on endoscopic camera

Abstract

The present invention provides an oral cavity recognition method based on an endoscopic camera, which semantically segments oropharyngeal images collected by an endoscope to obtain specific tissue images, and then accurately locates the corresponding parts in space. The present invention pre-labels a small number of collected oropharyngeal sample images, and uses an attention mechanism to label the associated pixels in the image of the region of interest to obtain a label set. The present invention efficiently performs inference calculations based only on prior knowledge of the spatial structure of oropharyngeal tissue; improves the neural network structure, and globally extracts and integrates the feature information of the oropharyngeal parts, thereby globally perceiving the correlation between different oropharyngeal parts, making the analysis results more accurate and effectively eliminating segmentation errors; adopts an improved single-label segmentation result to improve the accuracy of the overall output segmentation result, which can meet the requirements of oral sampling for clinical identification and positioning accuracy.

Description

Technical Field

[0001] The invention relates to an oral cavity recognition method based on an endoscopic camera, and belongs to the technical field of biomedical equipment. Background Art

[0002] Currently, oral cavity recognition is primarily achieved through visual inspection and dental mirrors. However, neither visual nor dental mirror methods can be applied to robots. The current lack of a robot-friendly oral cavity recognition method makes it impossible to use robots to guide medical staff or automated equipment to effectively treat different parts of the mouth. Summary of the Invention

[0003] The technical problem to be solved by the present invention is that there is currently a lack of a dedicated recognition and processing method for the oral cavity that can be applied to robots.

[0004] In order to solve the above technical problems, the technical solution of the present invention is to provide an oral cavity recognition method based on an endoscopic camera, which identifies and locates the oral cavity through an endoscopic camera, and is characterized by comprising the following steps:

[0005] Step 1: Obtain the original oropharyngeal image through the endoscope camera, and at the same time, obtain the segmentation ground truth map after a small amount of annotation;

[0006] Step 2: For the preprocessed original oropharyngeal image, a graph neural network is used to generate a coarse localization structure analysis map and calculate the segmentation sub-masks of each oropharyngeal feature area;

[0007] Step 3: Based on the hierarchical organizational knowledge of the oropharynx prior, the semantic categories and label hierarchy information are defined, and the adjacency transformation matrix is ​​further defined as the mask of the input graph neural network structure.

[0008] Step 4: Using the segmentation sub-masks of each oropharyngeal feature area, as well as the semantic categories and label hierarchical deconstruction structure information defined in step 3, the high-dimensional coarse parsed graph is embedded as a graph feature with spatial correlation.

[0009] Step 5: Using the graph features obtained in step 4 and the adjacency transformation matrix obtained in step 3, the global semantic information is further inferred and propagated through the graph neural network structure to extract new decomposition graph features;

[0010] Step 6: Using the new decomposition features obtained in step 5 and the segmentation sub-masks of each oropharyngeal feature area, we obtain the intermediate layer structure analysis map, which is then fused with the coarse positioning structure analysis map obtained in step 2 to obtain the fine positioning structure analysis map.

[0011] Step 7: Using the preprocessed original human body image, repeat steps 2, 4, 5, and 6 in the encoding and decoding structure of the graph neural network to train and form the final oropharyngeal key structure analysis model;

[0012] Step 8: Input the oropharyngeal image to be processed into the final oropharyngeal key structure analysis model to obtain the required precise analysis diagram.

[0013] Preferably, in step 1, the original oropharyngeal image is preprocessed.

[0014] Preferably, the pretreatment comprises the following steps:

[0015] Step 101: Perform different illumination and contrast processing on the original oropharyngeal image, and perform image transformation operations;

[0016] Step 102: Perform color normalization and resize the image to a uniform size;

[0017] Step 103: Perform data augmentation operations to flip the left and right sides of the image to achieve correct analysis of the positions of the left and right parts.

[0018] Preferably, the step 2 specifically includes the following steps:

[0019] The oropharyngeal image is input into the graph neural network, and the feature map is obtained after processing by the convolutional layer in the graph neural network; the spatial convolution pooling pyramid layer in the graph neural network performs hole convolution and parallel sampling on the feature map at different sampling rates, capturing the contextual information of the feature map at different scales to generate a coarse positioning structure analysis map.

[0020] Preferably, the specific mechanism of the graph neural network is as follows: given a one-dimensional embedding sequence as input, a pure Transformer-based encoder is used for feature representation. The Transformer encoder consists of a multi-head self-attention module and a multi-layer perceptron. In the first layer, the input of the self-attention is a three-dimensional tuple (query, key, value) calculated from the input:

[0021] query=Z l-1 W Q

[0022] key=Z l-1 W K

[0023] value=Z l-1 W V

[0024] Among them, W Q 、W K 、W Vare the learnable parameters of the three linear projection layers, Z l-1 represents the output of the l-1th layer of the Transformer encoder;

[0025] Self-attention is expressed as:

[0026]

[0027] Multi-head self-attention is obtained by concatenating multiple self-attention operations and then projecting them. The output of the multi-head self-attention is then transformed by a multi-layer perceptron block with residual skip connections:

[0028] Z l =MSA(Z l-1 )+MLP(MSA(Z l-1 ))∈R L×C

[0029] Thus, the output of each layer of Transformer is obtained: {Z 1 ,Z 2 ,…Z Le}, where MSA (multi-head self-attention) means using multiple queries to select multiple pieces of information from the input information in parallel, with each attention focusing on a different part of the input information, and then concatenating them. MLP (Multilayer Perceptron) represents a multilayer perceptron that maps a set of input vectors to a set of output vectors.

[0030] This paper provides a method for spatial localization of oropharyngeal tissue using an improved attention mechanism based on a graph neural network. This method semantically segments oropharyngeal images acquired through endoscopy to obtain specific tissue images, allowing for precise spatial localization of the corresponding parts. The method pre-annotates a small number of sample oropharyngeal images and uses an attention mechanism to annotate the associated pixels in the region of interest (ROI) image to generate an annotation set.

[0031] The present invention defines a graph-structured neural network for the image region of interest in images captured by an endoscopic camera, embeds high-dimensional encoding features into low-dimensional graph neural network features, and uses an improved attention mechanism to perform inference calculations. The generated new graph is then re-decoded to obtain a refined parsing graph, thereby efficiently training the entire neural network model to obtain the final semantic segmentation result and obtaining the spatial position information of the corresponding tissue based on the segmentation result. The present invention efficiently performs inference calculations based solely on prior knowledge of the spatial structure of oropharyngeal tissue; improves the neural network structure to globally extract and integrate oropharyngeal feature information, thereby globally perceiving the correlation between different oropharyngeal regions, resulting in higher accuracy of the parsing result and effectively eliminating segmentation errors; and uses an improved single-label segmentation result to improve the accuracy of the overall output segmentation result, which can meet the clinical identification and positioning accuracy requirements of oral sampling. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The present invention is a flowchart of an oral cavity recognition method based on an endoscopic camera disclosed in an embodiment. DETAILED DESCRIPTION

[0033] Below in conjunction with specific embodiment, further set forth the present invention.Should be understood that these embodiments are only used to illustrate the present invention and are not used in limiting the scope of the present invention.In addition, should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall equally within the scope limited by the appended claims of the application.

[0034] This embodiment provides an oral cavity recognition method based on an endoscopic camera, which recognizes and locates the oral cavity through an endoscopic camera.

[0035] Step 1: Obtain raw oropharyngeal images using an endoscopic camera. These images are annotated and used as samples for the next step of graph neural network training. Preprocess the raw oropharyngeal images, including but not limited to image enhancement, noise filtering, region of interest extraction, and image rotation and translation.

[0036] In this embodiment, preprocessing the original oropharyngeal image further includes the following steps:

[0037] Step 101: The original oropharyngeal image is processed with different illumination and contrast, and transformation operations such as image stretching and selection are performed. These operations improve the generalization performance of the graph neural network used subsequently.

[0038] Step 102: Perform color normalization on the original oropharyngeal image and resize the image to a uniform size.

[0039] Step 103: In order to better analyze different directions, data enhancement is required to flip the left and right sides of the image to achieve the correct analysis of the positions of the left and right parts.

[0040] Step 2: For the preprocessed original oropharyngeal image, use the graph neural network to generate a coarse positioning structure analysis map and calculate the segmentation sub-mask of the oropharyngeal feature parts. The specific steps include:

[0041] The preprocessed raw oropharyngeal image is input into the graph neural network, where it is processed by the convolutional layers to generate a feature map. The spatial convolutional pooling pyramid layer in the graph neural network performs dilated convolutions and parallel sampling on the feature map at different sampling rates, capturing contextual information at varying scales to generate a coarse parsed map. This coarse parsed map, which contains rich feature information, is then fed into the next network structure.

[0042] The specific mechanism of a graph neural network is as follows: given a one-dimensional embedding sequence as input, a pure Transformer-based encoder is used for feature representation. The Transformer encoder consists of a multi-head self-attention module and a multi-layer perceptron. In the first layer, the self-attention input is a three-dimensional tuple (query, key, value) calculated from the input:

[0043] query=Z l-1 W Q

[0044] key=Z l-1 W K

[0045] value=Z l-1 W V

[0046] Among them, W Q 、W K 、W V are the learnable parameters of the three linear projection layers, Z l-1 represents the output of the l-1th layer of the Transformer encoder. Self-attention can be expressed as:

[0047]

[0048] Multi-head self-attention is obtained by concatenating multiple self-attention operations and then projecting them. The output of the multi-head self-attention is then transformed by a multi-layer perceptron block with residual skip connections:

[0049] Z l =MSA(Z l-1 )+MLP(MSA(Z l-1 ))∈RL×C

[0050] Thus, the output of each layer of Transformer is obtained: MSA (multi-head self-attention) uses multiple queries to select multiple pieces of information from the input in parallel. Each attention segment focuses on a different part of the input, and then concatenates the information. MLP (Multilayer Perceptron) maps a set of input vectors to a set of output vectors.

[0051] Graph neural networks are based on multi-level feature fusion. This paper represents features from multiple layers of the Transformer encoder, with each layer focusing on a specific selected Transformer layer. After pixel-level feature fusion, a convolutional layer is used again. Finally, the feature maps obtained from each path are concatenated in the channel dimension and upsampled using bilinear interpolation to obtain the final output.

[0052] Step 3: Based on the hierarchical organizational knowledge of the oropharynx prior, the semantic categories and label hierarchy information are defined, and the adjacency transformation matrix is ​​further defined as the mask of the input graph neural network structure.

[0053] Step 4: Using the segmentation sub-masks of each oropharyngeal feature area, as well as the semantic categories and label hierarchical deconstruction structure information defined in step 3, the high-dimensional coarse parsed graph is embedded as a graph feature with spatial correlation.

[0054] Step 5: Using the graph features obtained in step 4 and the adjacency transformation matrix obtained in step 3, the global semantic information is further inferred and propagated through the graph neural network structure to extract new decomposition graph features.

[0055] Step 6: Use the new decomposition features obtained in step 5 and the segmentation masks of each oropharyngeal feature area to fuse them with the coarse positioning structure analysis map obtained in step 2 to obtain a fine positioning structure analysis map;

[0056] Step 7: Using the preprocessed original human body image, repeat the above steps 2, 4, 5, and 6 in the encoding and decoding structure of the graph neural network for training to form the final oropharyngeal key structure analysis model.

[0057] Step 8: Input the oropharyngeal image to be processed into the final oropharyngeal key structure analysis model to obtain the required precise analysis diagram.

Claims

1. An oral cavity recognition method based on an endoscopic camera, which identifies and locates the oral cavity through an endoscopic camera, characterized in that: The following steps are involved: Step 1: Obtain the original oropharyngeal image through the endoscope camera, and at the same time, obtain the segmentation ground truth map after a small amount of annotation; Step 2: For the preprocessed original oropharyngeal image, a graph neural network is used to generate a coarse localization structure analysis map and calculate the segmentation sub-masks of each oropharyngeal feature area; Step 3: Based on the oropharyngeal prior knowledge of hierarchical structure, semantic categories and label hierarchy information are defined, and the adjacency transformation matrix is ​​further defined as the mask of the input graph neural network structure; Step 4: Using the segmentation sub-masks of each oropharyngeal feature area, as well as the semantic categories and label hierarchical decomposition structure information defined in step 3, the high-dimensional coarse parsed graph is embedded into a graph feature with spatial correlation. Step 5: Using the graph features obtained in step 4 and the adjacency transformation matrix obtained in step 3, the global semantic information is further inferred and propagated through the graph neural network structure to extract new decomposition graph features; Step 6: Using the new decomposition features obtained in step 5 and the segmentation sub-masks of each oropharyngeal feature area, we obtain the intermediate layer structure analysis map, which is then fused with the coarse positioning structure analysis map obtained in step 2 to obtain the fine positioning structure analysis map. Step 7: Using the preprocessed original human body image, repeat steps 2, 4, 5, and 6 in the encoding and decoding structure of the graph neural network to train and form the final oropharyngeal key structure analysis model; Step 8: Input the oropharyngeal image to be processed into the final oropharyngeal key structure analysis model to obtain the required precise analysis diagram.

2. The oral cavity recognition method based on an endoscopic camera according to claim 1, characterized in that: In step 1, the original oropharyngeal image is preprocessed.

3. The oral cavity recognition method based on an endoscopic camera according to claim 2, characterized in that: The pretreatment comprises the following steps: Step 101: Perform different illumination and contrast processing on the original oropharyngeal image, and perform image transformation operations; Step 102: Perform color normalization and resize the image to a uniform size; Step 103: Perform data augmentation operations to flip the left and right sides of the image to achieve correct analysis of the positions of the left and right parts.

4. The oral cavity recognition method based on an endoscopic camera according to claim 1, characterized in that: The step 2 specifically includes the following steps: The oropharyngeal image is input into the graph neural network, and the feature map is obtained after processing by the convolutional layer in the graph neural network; the spatial convolution pooling pyramid layer in the graph neural network performs hole convolution and parallel sampling on the feature map at different sampling rates, capturing the contextual information of the feature map at different scales to generate a coarse positioning structure analysis map.

5. The oral cavity recognition method based on an endoscopic camera according to claim 4, characterized in that: The specific mechanism of the graph neural network is as follows: given a one-dimensional embedding sequence as input, a pure Transformer-based encoder is used for feature representation. The Transformer encoder consists of a multi-head self-attention module and a multi-layer perceptron. In the first layer, the input of the self-attention is a three-dimensional tuple (query, key, value) calculated from the input: query=Z l-1 W Q key=Z l-1 W K value=Z l-1 W V Among them, W Q 、W K 、W V are the learnable parameters of the three linear projection layers, Z l-1 represents the output of the l-1th layer of the Transformer encoder; Self-attention is expressed as: Multi-head self-attention is obtained by concatenating multiple self-attention operations and then projecting them. The output of the multi-head self-attention is then transformed by a multi-layer perceptron block with residual skip connections: Z l =MSA(Z l-1 )+MLP(MSA(Z l-1 ))∈R L×C Thus, the output of each layer of Transformer is obtained: Among them, MSA means using multiple queries to select multiple information from the input information in parallel. Each attention focuses on different parts of the input information, and then splices them together. MLP means multi-layer perceptron, which maps a set of input vectors to a set of output vectors.

Images