Multi-region identification method, device and equipment of endoscope image and storage medium

Abstract

The application relates to a multi-region identification method, device and equipment of an endoscope image and a storage medium, wherein the multi-region identification method of the endoscope image comprises the following steps: projecting a three-dimensional anatomical atlas matched with an endoscope image to the endoscope image, and generating a region mask of the endoscope image according to a projection result; wherein the region mask is used for indicating an anatomical region corresponding to each pixel position in the endoscope image; based on the region mask, a constraint mask of each anatomical region is generated; based on a first image feature of the endoscope image and the region mask, a target image feature of the endoscope image is determined; the target image feature is enhanced based on the constraint mask of each anatomical region, and region identification is performed on the endoscope image based on the enhanced target image feature. Through the application, the problem that accurate identification of multiple anatomical regions in an endoscope image is difficult to realize is solved, and accurate identification of multiple anatomical regions in an endoscope image is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a multi-region identification method, device and equipment of an endoscope image and a storage medium. BACKGROUND

[0002] In modern medical diagnosis, endoscopy is a key means for detecting diseases in the throat and other parts. However, its image analysis is highly dependent on the clinical experience of doctors, and faces challenges such as complex and variable anatomical structures, blurred boundaries of diseased tissues, and easy missed diagnosis of small lesions.

[0003] To solve the above problems, the existing method often uses a pre-trained image recognition model to identify the collected endoscope image, but such a model still has the problems of insufficient recognition accuracy and poor robustness, and it is difficult to achieve accurate identification of multiple anatomical regions in the endoscope image.

[0004] To solve the problem that it is difficult to achieve accurate identification of multiple anatomical regions in the endoscope image in the related art, no effective solution has been proposed so far. SUMMARY

[0005] A multi-region identification method, device, equipment and storage medium of an endoscope image are provided in the present embodiment to solve the problem that it is difficult to achieve accurate identification of multiple anatomical regions in the endoscope image in the related art.

[0006] In a first aspect, a multi-region identification method of an endoscope image is provided in the present embodiment, comprising:

[0007] projecting a three-dimensional anatomical atlas matched with the endoscope image to the endoscope image, and generating a region mask of the endoscope image according to the projection result; wherein the region mask is used to indicate the anatomical region corresponding to each pixel position in the endoscope image;

[0008] generating a constraint mask of each anatomical region based on the region mask;

[0009] determining a target image feature of the endoscope image based on a first image feature of the endoscope image and the region mask;

[0010] enhancing the target image feature based on the constraint mask of each anatomical region, and performing region identification on the endoscope image based on the enhanced target image feature.

[0011] In some embodiments, the projecting a three-dimensional anatomical atlas matched with the endoscope image to the endoscope image, and generating a region mask of the endoscope image according to the projection result comprises:

[0012] establish a correspondence between the endoscope image and the three-dimensional anatomical atlas based on a plurality of feature points in the endoscope image;

[0013] project vertices of each anatomical region patch in the three-dimensional anatomical atlas to the endoscope image using the correspondence to obtain a projected two-dimensional point set;

[0014] determine the anatomical region corresponding to each pixel position in the endoscope image according to the projected two-dimensional point set to generate the region mask.

[0015] In some embodiments, the generating of the constraint mask of each anatomical region based on the region mask comprises:

[0016] determining a core region corresponding to each anatomical region; the core region is a set of pixels in the region mask belonging to the corresponding anatomical region;

[0017] determining a transition region corresponding to each anatomical region; each pixel in the transition region is less than or equal to a preset threshold distance from the boundary of the anatomical region;

[0018] performing maximum fusion of the mask of each core region and the mask of the corresponding transition region to generate the constraint mask of the corresponding anatomical region.

[0019] In some embodiments, after the generating of the constraint mask of each anatomical region based on the region mask, the method further comprises:

[0020] determining a plurality of reflective regions in the endoscope image, and a modulation function matching the reflective intensity of each reflective region;

[0021] performing convolution operation on image blocks corresponding to different modulation functions in the endoscope image according to the modulation functions to obtain second image features of the endoscope image;

[0022] optimizing the second image features based on preset parameters to obtain the first image features of the endoscope image; the preset parameters include reflectivity and compensation intensity coefficient.

[0023] In some embodiments, the determining of the target image features of the endoscope image based on the first image features of the endoscope image and the region mask comprises:

[0024] inputting the first image features and the region mask into a pre-constructed double-branch network for processing to obtain first target features and second target features of the endoscope image; the double-branch network comprises a first branch and a second branch;

[0025] wherein the first branch is configured to splice the first image feature with the region mask to obtain the first target feature; and the second branch is configured to determine the second target feature based on the first image feature and a local morphological feature of a three-dimensional mucosa surface; and the three-dimensional mucosa surface is reconstructed based on the endoscopic image;

[0026] fusing the first target feature and the second target feature of the endoscopic image to obtain a target image feature.

[0027] In some embodiments, the enhancing the target image feature based on the constraint mask comprises:

[0028] determining a region saliency score of each of the anatomical regions based on the target image feature;

[0029] adaptively enhancing the target image feature according to the region saliency score of each of the anatomical regions, the constraint mask, and a preset enhancement factor.

[0030] In some embodiments, the performing region recognition on the endoscopic image based on the enhanced target image feature comprises:

[0031] inputting the enhanced target image feature into a convolutional network for processing to output an anatomical region segmentation result of the endoscopic image.

[0032] In a second aspect, an embodiment of the present disclosure provides an endoscopic image multi-region recognition device, comprising:

[0033] a projection module configured to project a three-dimensional anatomical atlas matched with the endoscopic image to the endoscopic image to generate a region mask of the endoscopic image according to a projection result; wherein the region mask is configured to indicate an anatomical region corresponding to each pixel position in the endoscopic image;

[0034] a generation module configured to generate a constraint mask of each of the anatomical regions based on the region mask;

[0035] a fusion module configured to determine a target image feature of the endoscopic image based on a first image feature of the endoscopic image and the region mask;

[0036] a recognition module configured to enhance the target image feature based on the constraint mask of each of the anatomical regions, and perform region recognition on the endoscopic image based on the enhanced target image feature.

[0037] In a third aspect, a computer device is provided in the embodiments, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-region identification method of the endoscopic image according to the first aspect when executing the computer program.

[0038] In a fourth aspect, a storage medium is provided in the embodiments, and the storage medium stores a computer program, and the computer program is executable on a processor to implement the multi-region identification method of the endoscopic image according to the first aspect.

[0039] Compared with the related art, the multi-region identification method of the endoscopic image, the device, the equipment and the storage medium provided in the embodiments project a three-dimensional anatomical atlas matched with the endoscopic image to the endoscopic image, generate a region mask of the endoscopic image according to a projection result, wherein the region mask is used to indicate an anatomical region corresponding to each pixel position in the endoscopic image, generate a constraint mask of each anatomical region based on the region mask, determine a target image feature of the endoscopic image based on a first image feature of the endoscopic image and the region mask, enhance the target image feature based on the constraint mask of each anatomical region, and perform region identification on the endoscopic image based on the enhanced target image feature, thereby solving the problem that it is difficult to accurately identify multiple anatomical regions in the endoscopic image, and achieving accurate identification of multiple anatomical regions in the endoscopic image.

[0040] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0041] The drawings described herein are intended to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute an improper limitation of the present application. In the drawings:

[0042] Figure 1 is a flowchart of the multi-region identification method of the endoscopic image provided in an embodiment of the present application;

[0043] Figure 2 is a flowchart of the region mask generation method provided in an embodiment of the present application;

[0044] Figure 3 is a flowchart of the constraint mask generation method provided in an embodiment of the present application;

[0045] Figure 4 is a flowchart of the cross-scale image feature fusion method provided in an embodiment of the present application;

[0046] Figure 5is a flowchart of an image fusion feature enhancement method provided by an embodiment of the present application;

[0047] Figure 6 is a flowchart of a multi-anatomical region identification method provided by an embodiment of the present application;

[0048] Figure 7 is a structural block diagram of a multi-region identification device for endoscopic images provided by an embodiment of the present application.

[0049] In the figure: 10, projection module; 20, generation module; 30, fusion module; 40, identification module. DETAILED DESCRIPTION

[0050] In order to more clearly understand the purpose, technical scheme and advantages of the present application, the present application is described and explained below in conjunction with the drawings and embodiments.

[0051] Unless otherwise defined, technical terms or scientific terms involved in the present application shall have the general meaning understood by those skilled in the art with general knowledge. In the present application, "one", "a", "an", "the", "these" and similar words do not represent a quantitative limitation, and they can be singular or plural. In the present application, the terms "include", "contain", "have" and any variants thereof have the purpose of covering non-exclusive inclusion; for example, a process, method and system, product or device containing a series of steps or modules (units) are not limited to the listed steps or modules (units), but can include steps or modules (units) not listed, or can include other steps or modules (units) inherent to the process, method, product or device. In the present application, the terms "connected", "connected", "coupled" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. In the present application, "multiple" means two or more. The association between the associated objects is described by the term "and / or", which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. In general, the character " / " represents an "or" relationship between the objects before and after. In the present application, the terms "first", "second", "third" and the like are only used to distinguish similar objects, and do not represent a specific order of the objects.

[0052] In the present embodiment, a multi-region identification method for endoscopic images is provided, Figure 1 is a flowchart of the multi-region identification method for endoscopic images of the present embodiment, as Figure 1 shown, the flowchart includes the following steps:

[0053] Step S110, projecting the three-dimensional anatomical atlas matched with the endoscope image to the endoscope image, and generating a region mask of the endoscope image according to a projection result; wherein the region mask is used to indicate an anatomical region corresponding to each pixel position in the endoscope image;

[0054] Specifically, medical scan data of a target site (such as a throat, a digestive tract organ, etc.) of a plurality of healthy individuals is acquired, including Computed Tomography (CT) data / Nuclear Magnetic Resonance Imaging (NMR) data, etc. Subsequently, a three-dimensional surface reconstruction algorithm such as Marching Cubes (MC) or Marching Tetrahedra (MT) is used to process the acquired medical scan data to generate a corresponding three-dimensional surface mesh model, which is composed of a plurality of surface mesh patches. The surface mesh model obtained by reconstruction is labeled to identify a plurality of key regions (such as the epiglottis, vocal cords, and piriform fossa of the throat) of the target site. The labeling process can be automatically completed by a target detection or segmentation model based on deep learning, or manually labeled by a professional. Finally, the labeled three-dimensional surface mesh model is integrated and constructed into a three-dimensional anatomical atlas of the target site.

[0055] Further, an endoscope image to be recognized is acquired, and a corresponding three-dimensional anatomical atlas is matched according to the acquisition site. The spatial coordinates of the matched three-dimensional anatomical atlas are projected to the endoscope image plane, and a region mask of the endoscope image is generated according to the projection result. The mask accurately identifies the anatomical structure (such as the epiglottis, vocal cords, and piriform fossa) to which each pixel point in the image belongs, providing structured prior information for subsequent anatomical positioning and structure recognition, that is, helping to constrain the network to perform segmentation only in the anatomically feasible region, thereby improving the region recognition rate.

[0056] Step S120, generating a constraint mask of each anatomical region based on the region mask;

[0057] Specifically, a core region corresponding to each anatomical region is determined, and the core region is a set of pixels belonging to the corresponding anatomical region in the region mask. At the same time, a transition region corresponding to each anatomical region is determined, and the distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold.

[0058] Further, the mask of each core region and the mask of the corresponding transition region are fused to generate a constraint mask of the corresponding anatomical region, so as to embed the three-dimensional anatomical atlas as a spatial constraint and establish a topological relationship constraint using the anatomical structure information in the atlas.

[0059] At step S130, a target image feature of the endoscopic image is determined based on the first image feature of the endoscopic image and the region mask;

[0060] The first image feature of the endoscopic image is extracted, and then the target image feature of the endoscopic image is generated in combination with the region mask. The specific implementation includes splicing and fusing the first image feature of the endoscopic image and the region mask to obtain the target image feature, or inputting the first image feature of the endoscopic image and the region mask into a double-branch network, wherein the first branch generates a first target feature by splicing the first image feature and the region mask, and the second branch generates a second target feature in combination with the first image feature and a three-dimensional mucosal surface feature reconstructed based on the endoscopic image, and finally fusing the output features of the two branches to obtain the target image feature. In addition, the endoscopic image can be pre-processed (such as reflection compensation, noise suppression, etc.), and the subsequent processing is based on the optimized endoscopic image feature to improve the quality of the generated feature.

[0061] At step S140, the target image feature is enhanced based on the constraint mask of each anatomical region, and the endoscopic image is regionally recognized based on the enhanced target image feature.

[0062] Specifically, the target image feature is enhanced using the constraint mask of each anatomical region to obtain the enhanced target image feature. In other embodiments, the target image feature can be adaptively enhanced in combination with the region saliency score and the constraint mask of each anatomical region to effectively strengthen the key region feature and suppress the influence of non-key or interference regions, thereby improving the accuracy and robustness of subsequent image region recognition.

[0063] Further, the enhanced target image feature is input into a selected image recognition model (such as a convolutional neural network, a Transformer architecture, or a graph neural network, etc.) for processing to obtain an anatomical region segmentation result of the endoscopic image. For example, the enhanced target image feature is input into a convolutional network to extract local anatomical structure features layer by layer, and the same size of convolution kernel is used at each layer to ensure the consistency of the receptive field, and finally the anatomical region segmentation result of the endoscopic image is output. The anatomical region segmentation result accurately identifies a plurality of specific anatomical structures contained in the current collection site, such as the epiglottis, vocal cords, and piriform fossa of the throat, the fundus, body, and antrum of the stomach, etc.

[0064] It should be noted that the endoscopic image in the present embodiment can be derived from a public medical image database, medical teaching materials, algorithm development and verification data sets, or device test simulation environment, etc., which are not specifically limited here. Correspondingly, the region recognition result of the endoscopic image described above can be used for statistical analysis of medical image data (such as image feature distribution statistics, etc.), training and verification of related image processing algorithms, testing of medical image analysis software, etc.

[0065] In modern medical diagnosis, endoscopy is a crucial tool for detecting diseases in areas such as the throat. However, image analysis heavily relies on the physician's clinical experience and faces challenges such as complex and variable anatomical structures, blurred lesion boundaries, and the potential for missing small lesions. To address this issue, existing methods often employ pre-trained image recognition models to identify acquired endoscopic images. However, these models still suffer from insufficient recognition accuracy and poor robustness, making it difficult to accurately identify multiple anatomical regions within endoscopic images.

[0066] Compared to existing technologies, this application projects a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generates a region mask for the endoscopic image based on the projection result. The region mask indicates the anatomical region corresponding to each pixel position in the endoscopic image. Based on the region mask, a constraint mask for each anatomical region is generated. Based on the first image features of the endoscopic image and the region mask, the target image features of the endoscopic image are determined. The target image features are enhanced based on the constraint masks of each anatomical region, and region recognition is performed on the endoscopic image based on the enhanced target image features. Based on this, by introducing a three-dimensional anatomical atlas as a spatial constraint and utilizing the anatomical structure information in the atlas to establish topological relationship constraints, a feature representation combining anatomical priors is achieved, significantly improving the rationality and accuracy of region recognition. This solves the problem of accurately recognizing multiple anatomical regions in endoscopic images, thus realizing accurate recognition of multiple anatomical regions in endoscopic images.

[0067] In some of these embodiments, such as Figure 2 As shown, step S110, which involves projecting a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image and generating a region mask for the endoscopic image based on the projection result, includes the following steps:

[0068] Step S111: Based on multiple feature points in the endoscopic image, establish the correspondence between the endoscopic image and the three-dimensional anatomical atlas;

[0069] Step S112: Using the correspondence, project the vertices of each anatomical region in the three-dimensional anatomical atlas onto the endoscopic image to obtain the projected two-dimensional point set.

[0070] Step S113: Based on the projected two-dimensional point set, determine the anatomical region corresponding to each pixel position in the endoscopic image to generate a region mask.

[0071] Specifically, feature operators such as Scale-invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) are used to extract features from endoscopic images to obtain multiple feature points in the endoscopic images and establish the correspondence between each feature point in the endoscopic images and the anatomical structures in the three-dimensional anatomical atlas.

[0072] Using correspondences, the vertices of each anatomical region patch in the 3D anatomical atlas are projected onto the endoscopic image plane. Based on the projected 2D point set, the anatomical region corresponding to each pixel position in the endoscopic image is determined to generate a region mask. The region mask... The specific expression is as follows:

[0073] (1)

[0074] In equation (1), This represents the Euclidean distance from the anatomical region patch to the image acquisition camera; This represents the two-dimensional point set after projecting all vertices of the anatomical region k. For example, the pyriform fossa region has 120 vertices, which correspond to 120 two-dimensional points after projection. This involves calculating the convex hull. It's understandable that during mask generation, for each pixel in the image... Find the projected convex hull of all anatomical regions k. Select from the regions The smallest region (i.e., the nearest anatomical structure) is labeled with that region. .

[0075] In this embodiment, a correspondence between the endoscope image and the three-dimensional anatomical atlas is established based on multiple feature points in the endoscope image. Using the correspondence, the vertices of each anatomical region patch in the three-dimensional anatomical atlas are projected onto the endoscope image to obtain a projected two-dimensional point set. Based on the projected two-dimensional point set, the anatomical region corresponding to each pixel position in the endoscope image is determined to generate a region mask, thereby achieving accurate generation of the region mask.

[0076] In some of these embodiments, such as Figure 3 As shown, step S120, which generates a constraint mask for each anatomical region based on the region mask, includes the following steps:

[0077] Step S121: Determine the core region corresponding to each anatomical region; the core region is the set of pixels belonging to the corresponding anatomical region in the region mask;

[0078] Step S122: Determine the transition region corresponding to each anatomical region; the distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold.

[0079] Step S123: Maximize the mask of each core region and the mask of the corresponding transition region to generate the constraint mask of the corresponding anatomical region.

[0080] Specifically, based on the generated region mask, a core region is determined for each anatomical region. The core region is the set of pixels belonging to the corresponding anatomical region in the region mask. (Core region mask) The specific expression is as follows:

[0081] (2)

[0082] In equation (2), the pixel mask value of a pixel belonging to the anatomical region k is 1; otherwise, the pixel mask value of a pixel not belonging to the anatomical region k is 0.

[0083] Based on the generated region mask, a transition region is determined for each anatomical region. The distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold. (Transition region mask) The specific expression is as follows:

[0084] (3)

[0085] In equation (3), Indicates the boundary of the region; This represents a preset threshold, for example... Set to 15. If the distance between the pixel and the boundary of the anatomical region is less than or equal to the preset threshold. If the pixel value is 0, it belongs to the transition region and the mask value is 0.5; otherwise, the pixel mask value is 0. Here, the distance between the pixel and the boundary of the anatomical region is defined as the minimum Euclidean distance from the pixel to the boundary of the anatomical region.

[0086] Furthermore, the masks of each core region and the corresponding transition region are fused by maximizing their values ​​to generate the constraint mask for the corresponding anatomical region. The specific expression is as follows:

[0087] (4)

[0088] In equation (4), This represents the constraint mask for each anatomical region k; max() indicates taking the maximum value.

[0089] In this embodiment, a core region corresponding to each anatomical region is determined. The core region is the set of pixels belonging to the corresponding anatomical region in the region mask. A transition region corresponding to each anatomical region is also determined. The distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold. Finally, the mask of each core region and the mask of the corresponding transition region are fused by the maximum value to generate a constraint mask for the corresponding anatomical region. This allows for focusing on the anatomical region while taking into account the uncertainty of the edges in subsequent image processing, which helps to improve the robustness of region recognition and segmentation.

[0090] In some embodiments, after generating a constraint mask for each anatomical region based on the region mask, the above-described multi-region identification method for endoscopic images further includes the following steps:

[0091] Identify multiple reflective regions in the endoscopic image and a modulation function that matches the reflective intensity of each reflective region;

[0092] Based on each modulation function, the image blocks corresponding to different modulation functions in the endoscopic image are convolved to obtain the second image features of the endoscopic image;

[0093] The second image features are optimized based on preset parameters to obtain the first image features of the endoscope image; the preset parameters include reflectivity and compensation intensity coefficient.

[0094] Specifically, the endoscopic image is inspected to identify reflective areas. For example, pixels with a Y channel brightness value greater than 220 and a gradient change value less than 5 in the YUV color space are detected as reflective areas.

[0095] The reflectivity of each reflective area is analyzed, and the calculation methods include, but are not limited to, the average brightness intensity and brightness distribution intensity of the local reflective area. For example, the specific formula for calculating reflectivity intensity is as follows:

[0096] (5)

[0097] In equation (5), G represents the reflective area; This represents the pixel neighborhood selected by a 7x7 window; Indicates the number of pixels; The reflectivity is calculated by taking each pixel position (x, y) as the center and counting the proportion of pixels detected as reflective areas within its neighborhood window, which is the local reflectivity.

[0098] The filter operator that matches the reflectivity of each reflective region is determined, and its dynamic matching mechanism is as follows:

[0099] (6)

[0100] In equation (6), This represents the matched filter operator. If This indicates that the area is highly reflective. The Laplacian operator is used, leveraging its edge-enhancing properties to preserve the tissue structure. This indicates that the area is a moderately reflective area, and the following method is used. and combination, Used to detect vertical edges Used to detect horizontal edges; if This indicates that the area is a weakly reflective region. A Gaussian operator is used to smooth the mucosa and preserve its details. It should be noted that the above equation (6) is an example of a matching mechanism. In practical applications, the operator selection logic can be flexibly adjusted according to specific needs.

[0101] Furthermore, the corresponding modulation function is determined based on the matched filter operator, as shown in the following expression:

[0102] (7)

[0103] In equation (7), Represents the modulation function; This represents the initial weighting coefficients.

[0104] Based on the matched modulation functions, the image patches corresponding to different modulation functions in the endoscopic image are convolved to obtain the second image feature of the endoscopic image. The specific expression for the convolution operation is as follows:

[0105] (8)

[0106] In equation (8), This represents an image patch in an endoscopic image. Represents image blocks The corresponding modulation function; This represents the image features after the convolution operation, i.e., the second image features.

[0107] Next, the second image features are optimized based on preset parameters to obtain the first image features of the endoscopic image. The preset parameters include reflectance and compensation intensity coefficient. The specific expression for this optimization process is as follows:

[0108] (9)

[0109] In equation (9), This represents the optimized image features, i.e., the first image features; Indicates the compensation strength coefficient, for example Set to 0.05; Indicates reflectivity.

[0110] This embodiment identifies multiple reflective regions in an endoscopic image and a modulation function matching the reflective intensity of each region. Based on these modulation functions, image blocks corresponding to different modulation functions in the endoscopic image are convolved to obtain second image features. These second image features are then optimized based on preset parameters to obtain first image features. A dynamic kernel function is designed based on this, and by analyzing the illumination intensity distribution in local areas, the weight parameters of the convolution kernel are adaptively adjusted to compensate for gradients in the reflective regions. This scheme fundamentally suppresses feature interference from reflective regions and compensates for effective lesion features, achieving endoscopic image enhancement. It also overcomes the bottleneck of detail loss and structural distortion caused by general image restoration, contributing to improved accuracy in subsequent image region recognition.

[0111] In some of these embodiments, such as Figure 4 As shown, step S130, which determines the target image features of the endoscope image based on the first image features and region mask of the endoscope image, includes the following steps:

[0112] Step S131: Input the first image features and region mask into the pre-constructed dual-branch network for processing to obtain the first target features and the second target features of the endoscope image; the dual-branch network includes a first branch and a second branch;

[0113] The first branch is used to concatenate the first image features with the region mask to obtain the first target features; the second branch is used to determine the second target features based on the first image features and the local morphological features of the three-dimensional mucosal surface; the three-dimensional mucosal surface is reconstructed based on the endoscopic image.

[0114] Step S132: The first target feature and the second target feature of the endoscopic image are fused to obtain the target image feature.

[0115] Specifically, the first image features and region mask are input into a pre-constructed dual-branch network for processing to obtain the first target features and the second target features of the endoscopic image. The dual-branch network includes a first branch and a second branch.

[0116] The first branch is the macroscopic anatomy branch (processing global anatomical structures), used to concatenate the first image features with a region mask to obtain the first target feature; the second branch is the microscopic lesion branch (focusing on mucosal surface details), used to determine the second target feature based on the first image features and the local morphological features of the three-dimensional mucosal surface. The three-dimensional mucosal surface is reconstructed from endoscopic images, and its local morphological features include mucosal surface distance information, surface curvature features, etc.

[0117] Furthermore, the first and second target features of the endoscopic image are fused across scales to obtain the target image features.

[0118] For example, the first branch uses U-Net network encoding, and the second branch uses HRNet network encoding, with the specific expressions as follows:

[0119] (10)

[0120] (11)

[0121] In equation (10), ; Indicates the first image feature; This represents the region mask. In equation (11), Indicates the second target feature; Indicates the first image feature; This represents the mucosal surface distance map. It should be noted that the mucosal surface distance map is used to indicate the perpendicular distance (i.e., normal distance) from a point corresponding to image coordinates (x, y) on the reconstructed 3D mucosal surface to the optimal reference plane locally fitted near the (x, y) point.

[0122] The first and second target features are input into a cross-attention network for cross-scale feature fusion to obtain the target image features. The specific expression for this fusion process is as follows:

[0123] (12)

[0124] In equation (12), This represents the features of the target image obtained through fusion; ; This represents the second target feature.

[0125] In this embodiment, the first image features and region mask are input into a pre-constructed dual-branch network for processing to obtain the first target feature and the second target feature of the endoscopic image. The dual-branch network includes a first branch and a second branch. The first branch concatenates the first image features with the region mask to obtain the first target feature. The second branch determines the second target feature based on the first image features and the local morphological features of the three-dimensional mucosal surface, which is reconstructed from the endoscopic image. Then, the first and second target features of the endoscopic image are fused to obtain the target image features. This fusion of macroscopic anatomical branches and microscopic lesion branches enables cross-scale feature interaction, allowing simultaneous identification of macroscopic anatomical regions and microscopic lesions. By utilizing the constraints of anatomical atlases and multi-scale lesion perception mechanisms, the recognition rate of difficult sample regions is improved.

[0126] In some of these embodiments, such as Figure 5 As shown, step S140, which enhances the target image features based on constraint masks, includes the following steps:

[0127] Step S141: Based on the features of the target image, determine the regional salience score of each anatomical region;

[0128] Step S142: Adaptively enhance the target image features based on the regional saliency score, constraint mask, and preset enhancement factor for each anatomical region.

[0129] Specifically, based on the features of the target image, the saliency of each anatomical region is evaluated to obtain a saliency score for each anatomical region. The specific expression for this evaluation method is as follows:

[0130] (13)

[0131] In equation (13), significance is calculated using the differential rate of change of the feature. Wherein, The regional significance score represents the anatomical region k. Represents the set of all pixels in the anatomical region k; This represents the number of pixels in the anatomical region k; The gradient represents the feature of the target image.

[0132] Furthermore, based on the regional saliency score, constraint mask, and preset enhancement factor for each anatomical region, adaptive enhancement of the target image features is performed. The specific expression for this enhancement process is as follows:

[0133] (14)

[0134] In equation (14), This represents the enhanced features of the target image; Represents the features of the target image; This represents a preset enhancement factor, such as Set to 0.3; The regional significance score represents the anatomical region k. The constraint mask represents the anatomical region k.

[0135] In this embodiment, based on the features of the target image, the regional saliency score of each anatomical region is determined. Based on the regional saliency score of each anatomical region, the constraint mask, and the preset enhancement factor, the features of the target image are adaptively enhanced, thereby effectively strengthening the features of key regions and improving the accuracy and robustness of subsequent image region recognition.

[0136] In some of these embodiments, such as Figure 6 As shown, step S140, which involves region recognition of the endoscopic image based on the enhanced target image features, includes the following steps:

[0137] Step S143: The enhanced target image features are input into a convolutional network for processing to output the anatomical region segmentation results of the endoscopic image.

[0138] Specifically, the enhanced target image features are input into a convolutional network for processing. This network extracts local anatomical structural features from the image through layer-by-layer, multi-level convolutional operations, while maintaining the consistency of the receptive field to preserve key spatial structural information, ultimately outputting the anatomical region segmentation result of the endoscopic image. The specific expression of its recognition process is as follows:

[0139] (15)

[0140] In equation (15), This represents the enhanced target image features; Softmax() is the activation function; Conv() is the convolution operation; This indicates the results of the anatomical region segmentation.

[0141] The anatomical region segmentation results accurately identify multiple specific anatomical structures contained in the current sampling site. For example, in the pharynx, key structures such as the epiglottis, vocal cords, and pyriform fossa are distinguished and marked; in the stomach, the main anatomical regions such as the fundus, body, and antrum are clearly segmented.

[0142] In this embodiment, the enhanced target image features are input into a convolutional network for processing to output the anatomical region segmentation results of the endoscopic image, thereby achieving accurate multi-region recognition of the endoscopic image and providing an accurate basis for anatomical structure information.

[0143] The following describes and illustrates this embodiment through specific examples.

[0144] CT / MRI scan data of the nasopharynx and larynx were collected from 200 healthy volunteers in advance. The MarchingCubes algorithm was used to perform three-dimensional reconstruction of the CT / MRI scan data of the nasopharynx and larynx, generating a mesh patch of the pharyngeal surface. Key areas of the three-dimensional reconstruction results were labeled, including the nasal vestibule, nasal septum, inferior turbinate, middle turbinate, pharyngeal orifice of the Eustachian tube, roof of the nasopharynx, posterior wall of the oropharynx, vallecula, epiglottis, vocal cords, ventricular folds, pyriform fossa, etc., to generate a three-dimensional anatomical atlas of the nasopharynx and larynx.

[0145] The process involves acquiring an endoscopic image of the nasopharynx and larynx to be identified, retrieving a 3D anatomical atlas of the nasopharynx and larynx, extracting multiple feature points from the endoscopic image, establishing a correspondence between the endoscopic image and the 3D anatomical atlas based on these feature points, and then projecting the vertices of each anatomical region patch from the 3D anatomical atlas onto the endoscopic image using this correspondence to obtain a projected 2D point set. Next, based on the projected 2D point set, the anatomical region corresponding to each pixel position in the endoscopic image is determined to generate a region mask.

[0146] For each anatomical region, a core region and a transition region are determined. The core region is the set of pixels belonging to the corresponding anatomical region in the region mask. In the transition region, the distance between each pixel and the boundary of the anatomical region is less than or equal to a preset threshold. The masks of each core region and the corresponding transition region are then fused by maximizing the values ​​to generate a constraint mask for the corresponding anatomical region.

[0147] Furthermore, multiple reflective regions in the endoscopic image were detected, and the local reflective intensity of these regions was analyzed and calculated. A modulation function matching the reflective intensity of each region was selected. Based on each modulation function, image blocks corresponding to different modulation functions in the endoscopic image were convolved to obtain the second image features of the endoscopic image. The second image features were then optimized based on reflectivity and compensation intensity coefficients to finally obtain the first image features of the endoscopic image.

[0148] Next, the first image features and region mask are input into a pre-constructed dual-branch network for processing to obtain the first target features and the second target features of the endoscopic image. The dual-branch network includes a first branch and a second branch. The first branch uses U-Net network encoding to concatenate the first image features with the region mask to obtain the first target features. The second branch uses HRNet network to determine the second target features based on the first image features and three-dimensional mucosal surface distance information, where the three-dimensional mucosal surface is reconstructed from the endoscopic image. Subsequently, the first and second target features of the endoscopic image are input into a cross-attention network for cross-scale feature fusion to obtain the target image features.

[0149] The regional saliency of target image features is calculated using the differential rate of change of features, resulting in a regional saliency score for each anatomical region. Based on this score, constraint mask, and preset enhancement factor, the target image features are adaptively enhanced. The high-dimensional feature tensor, after multi-scale feature fusion and spatial attention enhancement, is input into a convolutional network. By extracting local anatomical features layer by layer while maintaining receptive field consistency, the network ultimately outputs segmentation results containing specific anatomical structures such as the piriform fossa and epiglottis, and can also identify lesion locations.

[0150] This embodiment deeply embeds the three-dimensional anatomical topological constraints and dynamic feature focusing mechanism of the nasopharynx and larynx, and adaptively compensates for the interference of the mirror reflection area, thereby achieving accurate synchronous identification of multi-scale anatomical regions and lesions in nasopharyngeal endoscopic images.

[0151] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0152] This embodiment also provides a multi-region recognition device for endoscopic images, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below refer to combinations of software and / or hardware that implement a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0153] Figure 7 This is a structural block diagram of the multi-region recognition device for endoscopic images in this embodiment, as shown below. Figure 7 As shown, the device includes:

[0154] The projection module 10 is used to project a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generate a region mask of the endoscopic image based on the projection result; wherein, the region mask is used to indicate the anatomical region corresponding to each pixel position in the endoscopic image;

[0155] The generation module 20 is used to generate a constraint mask for each anatomical region based on the region mask;

[0156] The fusion module 30 is used to determine the target image features of the endoscope image based on the first image features and region mask of the endoscope image;

[0157] The recognition module 40 is used to enhance the features of the target image based on the constraint mask of each anatomical region, and to perform region recognition of the endoscopic image based on the enhanced target image features.

[0158] The apparatus provided in this embodiment projects a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generates a region mask for the endoscopic image based on the projection result. The region mask indicates the anatomical region corresponding to each pixel position in the endoscopic image. Based on the region mask, a constraint mask for each anatomical region is generated. Based on the first image features of the endoscopic image and the region mask, the target image features of the endoscopic image are determined. The target image features are enhanced based on the constraint masks of each anatomical region, and based on the enhanced target image features, region recognition is performed on the endoscopic image. This solves the problem of accurately identifying multiple anatomical regions in pharyngeal endoscopic images, and achieves accurate identification of multiple anatomical regions in pharyngeal endoscopic images.

[0159] In some embodiments, the projection module 10 is further configured to establish a correspondence between the endoscope image and the three-dimensional anatomical atlas based on multiple feature points in the endoscope image; using the correspondence, project the vertices of each anatomical region patch in the three-dimensional anatomical atlas onto the endoscope image to obtain a projected two-dimensional point set; and determine the anatomical region corresponding to each pixel position in the endoscope image based on the projected two-dimensional point set to generate a region mask.

[0160] In some embodiments, the generation module 20 is further configured to determine the core region corresponding to each anatomical region; the core region is the set of pixels belonging to the corresponding anatomical region in the region mask; determine the transition region corresponding to each anatomical region; the distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold; and fuse the mask of each core region and the mask of the corresponding transition region by maximizing the value to generate the constraint mask of the corresponding anatomical region.

[0161] In some of these embodiments, Figure 7 Based on this, the device also includes an optimization module for determining multiple reflective regions in the endoscopic image and a modulation function matching the reflective intensity of each reflective region; according to each modulation function, convolving the image blocks corresponding to different modulation functions in the endoscopic image to obtain the second image feature of the endoscopic image; optimizing the second image feature based on preset parameters to obtain the first image feature of the endoscopic image; the preset parameters include reflectivity and compensation intensity coefficient; and determining the target image feature of the endoscopic image based on the first image feature and the region mask.

[0162] In some embodiments, the fusion module 30 is further configured to input the first image features and the region mask into a pre-constructed dual-branch network for processing to obtain a first target feature and a second target feature of the endoscopic image; the dual-branch network includes a first branch and a second branch; wherein the first branch is used to concatenate the first image features with the region mask to obtain the first target feature; the second branch is used to determine the second target feature based on the first image features and the local morphological features of the three-dimensional mucosal surface; the three-dimensional mucosal surface is reconstructed based on the endoscopic image; the first target feature and the second target feature of the endoscopic image are fused to obtain the target image feature.

[0163] In some embodiments, the identification module 40 is further configured to determine the regional saliency score of each anatomical region based on the target image features; and to adaptively enhance the target image features according to the regional saliency score of each anatomical region, the constraint mask, and the preset enhancement factor.

[0164] In some embodiments, the recognition module 40 is also used to input the enhanced target image features into a convolutional network for processing, so as to output the anatomical region segmentation result of the endoscopic image.

[0165] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0166] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0167] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0168] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0169] S1, Project the three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generate a region mask for the endoscopic image based on the projection result; wherein, the region mask is used to indicate the anatomical region corresponding to each pixel position in the endoscopic image;

[0170] S2, Based on the region mask, generate the constraint mask for each anatomical region;

[0171] S3, Based on the first image features and region mask of the endoscope image, determine the target image features of the endoscope image;

[0172] S4 enhances the target image features based on the constraint mask of each anatomical region, and performs region recognition on the endoscopic image based on the enhanced target image features.

[0173] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0174] Furthermore, in conjunction with the multi-region recognition method for endoscopic images provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the multi-region recognition methods for endoscopic images in the above embodiments.

[0175] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0176] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0177] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0178] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A method for multi-region recognition of endoscopic images, characterized in that, include: A three-dimensional anatomical atlas matching the endoscopic image is projected onto the endoscopic image, and a region mask of the endoscopic image is generated based on the projection result; wherein, the region mask is used to indicate the anatomical region corresponding to each pixel position in the endoscopic image; Based on the region mask, a constraint mask for each of the anatomical regions is generated; Determining target image features of the endoscope image based on a first image feature and a region mask includes: concatenating the first image feature with the region mask to obtain a first target feature; determining a second target feature based on the first image feature and local morphological features of a three-dimensional mucosal surface; the three-dimensional mucosal surface is reconstructed based on the endoscope image; and fusing the first target feature and the second target feature of the endoscope image to obtain the target image feature. The target image features are enhanced based on the constraint mask of each anatomical region, and the endoscopic image is region-based based on the enhanced target image features.

2. The multi-region recognition method for endoscopic images according to claim 1, characterized in that, The step of projecting a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generating a region mask for the endoscopic image based on the projection result, includes: Based on multiple feature points in the endoscopic image, a correspondence between the endoscopic image and the three-dimensional anatomical atlas is established; Using the aforementioned correspondence, the vertices of each anatomical region patch in the three-dimensional anatomical atlas are projected onto the endoscopic image to obtain a projected two-dimensional point set; Based on the projected two-dimensional point set, the anatomical region corresponding to each pixel position in the endoscopic image is determined to generate the region mask.

3. The multi-region recognition method for endoscopic images according to claim 1, characterized in that, The step of generating a constraint mask for each anatomical region based on the region mask includes: Determine the core region corresponding to each of the anatomical regions; the core region is the set of pixels in the region mask that belong to the corresponding anatomical region. Determine a transition region corresponding to each of the anatomical regions; the distance between each pixel in the transition region and the boundary of the anatomical region is less than or equal to a preset threshold. The mask of each core region and the mask of the corresponding transition region are fused by maximizing the value to generate the constraint mask corresponding to the anatomical region.

4. The multi-region recognition method for endoscopic images according to claim 1, characterized in that, After generating a constraint mask for each anatomical region based on the region mask, the method further includes: Identify multiple reflective regions in the endoscopic image, and a modulation function that matches the reflective intensity of each reflective region; According to each of the modulation functions, the image blocks corresponding to different modulation functions in the endoscope image are convolved to obtain the second image features of the endoscope image; The second image features are optimized based on preset parameters to obtain the first image features of the endoscope image; the preset parameters include reflectivity and compensation intensity coefficient.

5. The multi-region recognition method for endoscopic images according to claim 1 or claim 4, characterized in that, The step of concatenating the first image features with the region mask to obtain the first target features, and determining the second target features based on the first image features and the local morphological features of the three-dimensional mucosal surface, includes: The first image features and the region mask are input into a pre-constructed dual-branch network for processing to obtain the first target features and the second target features of the endoscope image; the dual-branch network includes a first branch and a second branch; The first branch is used to concatenate the first image feature with the region mask to obtain the first target feature; the second branch is used to determine the second target feature based on the first image feature and the local morphological features of the three-dimensional mucosal surface.

6. The multi-region recognition method for endoscopic images according to claim 1, characterized in that, The enhancement of the target image features based on the constraint mask includes: Based on the target image features, a regional salience score is determined for each of the anatomical regions; The target image features are adaptively enhanced based on the regional saliency score of each anatomical region, the constraint mask, and the preset enhancement factor.

7. The multi-region recognition method for endoscopic images according to claim 1, characterized in that, The process of performing region recognition on the endoscopic image based on the enhanced target image features includes: The enhanced target image features are input into a convolutional network for processing to output the anatomical region segmentation results of the endoscope image.

8. A multi-region recognition device for endoscopic images, characterized in that, include: A projection module is used to project a three-dimensional anatomical atlas matching the endoscopic image onto the endoscopic image, and generate a region mask of the endoscopic image based on the projection result; wherein, the region mask is used to indicate the anatomical region corresponding to each pixel position in the endoscopic image; A generation module is used to generate a constraint mask for each of the anatomical regions based on the region mask; A fusion module is used to determine the target image features of the endoscope image based on the first image features of the endoscope image and the region mask; The fusion module is further configured to concatenate the first image features with the region mask to obtain a first target feature; determine a second target feature based on the first image features and the local morphological features of the three-dimensional mucosal surface; the three-dimensional mucosal surface is reconstructed based on the endoscopic image; and fuse the first target feature and the second target feature of the endoscopic image to obtain the target image feature. The recognition module is used to enhance the features of the target image based on the constraint mask of each of the anatomical regions, and to perform region recognition on the endoscopic image based on the enhanced target image features.

9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the multi-region recognition method for endoscopic images according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the multi-region recognition method for endoscopic images according to any one of claims 1 to 7.

Images