An inspection image processing method based on multi-exposure fusion

Abstract

The application relates to the field of medical image processing, in particular to an examination image processing method based on multi-exposure fusion. The method comprises the following steps: collecting multiple groups of images with different exposure intensities under the same examination visual angle; segmenting throat key anatomical structures in each group of images to obtain initial boundary coordinates; generating a triangular mesh covering the key region in the range of the initial boundary coordinates as a feature unit; calculating brightness feature values including brightness average values and brightness gradient values for the feature unit; establishing the association of the feature units among multiple images according to the brightness average values, and selecting the clearest expression of each boundary based on the brightness gradient values, and finally fusing all the optimized boundaries to form a strengthened result. The method significantly improves the detail performance of highlight and shadow areas in the image, effectively reduces the work burden of multi-image comparison and analysis while ensuring the diagnostic accuracy.

Description

Technical Field

[0001] This application relates to the field of medical image processing, and more particularly to an examination image processing method based on multi-exposure fusion. Background Technology

[0002] Laryngeal examination is an important diagnostic technique in otolaryngology. It primarily uses a laryngoscopy device to observe the anatomical structures and tissue conditions within the larynx, providing direct evidence for the diagnosis of vocal cord lesions, glottic abnormalities, and laryngeal tumors. During laryngeal examination, due to the narrow space and complex geometry of the larynx, and the limitations of cold light illumination, significant uneven brightness often occurs. For example, some areas may experience high reflectivity due to direct cold light, leading to localized overexposure, while deeper areas of the larynx or areas obscured by light may be shadowed due to insufficient illumination. Both of these situations can result in the loss of information on key pathological features, making it difficult for a single exposure image to comprehensively and clearly display all the tissue details that require attention.

[0003] To improve the integrity of image information, existing technologies employ a technique of continuously acquiring multiple sets of images with different exposure intensities from the same examination perspective. This involves synchronously comparing complementary brightness information from multiple sets of images to cover features in different brightness regions as much as possible, overcoming the limitations of single exposure. However, existing technologies still have significant shortcomings in feature extraction and fusion analysis of these multiple sets of multi-exposure images: due to the lack of a structured processing framework, existing methods either rely on tedious pixel-by-pixel calculations or struggle to accurately establish the correlation between corresponding anatomical structures in different exposure images. This not only significantly increases the workload and reduces processing efficiency in the entire analysis process but also easily leads to inaccurate boundary localization when extracting the boundaries of key larynx anatomical structures due to feature matching deviations between different exposure images. Consequently, it is difficult to efficiently and accurately generate processing results that clearly show the boundaries of key larynx anatomical structures, failing to meet the dual requirements of image quality and processing efficiency in clinical diagnosis. Summary of the Invention

[0004] To address the aforementioned problems, this application provides an inspection image processing method based on multi-exposure fusion. The method provided by this application includes the following steps:

[0005] Multiple sets of images with different exposure intensities are acquired from the same inspection perspective;

[0006] The key anatomical structures of the larynx in each group of images are segmented to obtain the initial boundary coordinates of each group of images;

[0007] Within the range of the initial boundary coordinates, a triangular mesh covering the key region is generated using the Deloni triangulation algorithm, wherein each triangular mesh is defined as a feature cell;

[0008] For each feature unit, calculate its average brightness value and brightness gradient value, thereby generating a brightness feature value containing the average brightness value and brightness gradient value for each feature unit in each group of images.

[0009] Based on the variation of the average brightness of multiple triangular mesh units corresponding to the same anatomical structure in different images, a corresponding correlation relationship of feature units between multiple sets of images is established.

[0010] For each initial boundary coordinate point, based on the brightness gradient value of its neighboring feature units in multiple sets of images with established associations, the optimal image of the boundary coordinate point is selected from multiple sets of images, and the position of the boundary coordinate point is corrected to the corresponding position in the optimal image.

[0011] By fusing all the corrected boundary coordinates, the final reinforced boundary of the key anatomical structures of the larynx is generated.

[0012] Specifically, the step of segmenting the key anatomical structures of the laryngeal passage in each set of images includes:

[0013] Before the step of semantic segmentation of key anatomical structures of the larynx in each group of images, image registration is performed on the multiple groups of images with different exposure intensities.

[0014] The key anatomical structures of the larynx, including the vocal cords, glottis, and arytenoid cartilage, were segmented using a deep learning model based on the U-Net architecture.

[0015] Specifically, the deep learning model based on the U-Net architecture incorporates an attention mechanism module during training and uses a dataset of laryngoscope images annotated by experts, employing data augmentation methods such as elastic deformation, random rotation, and brightness adjustment.

[0016] Specifically, the method for generating a triangular mesh covering the key region using the Deloni triangulation algorithm includes:

[0017] Uniform or image gradient-based adaptive feature point sampling is performed within the critical anatomical region enclosed by the initial boundary coordinates.

[0018] All boundary coordinate points and internal sampling points are merged into an input point set, and the Deloni triangulation algorithm is performed on the input point set to generate a triangular mesh.

[0019] Specifically, the method for calculating its average brightness and brightness gradient values ​​includes:

[0020] Extract the V channel values ​​of all pixels within the area covered by each triangular mesh feature unit in the HSV color space, and calculate their arithmetic mean to obtain the average brightness value;

[0021] The Sobel operator is used to calculate the gradient magnitude of all pixels in the region, and the average gradient value is obtained to obtain the brightness gradient value.

[0022] Specifically, establishing the correspondence between feature units in multiple sets of images based on the variation of the average brightness of multiple triangular mesh units corresponding to the same anatomical structure in different images includes:

[0023] For any triangular mesh cell in any image, candidate cells are found in another image based on their spatial topological location. The correlation coefficient of the average brightness sequence of the two cells in different exposure images is calculated, and the candidate cell with the largest correlation coefficient is established as the corresponding associated cell.

[0024] Specifically, the method for selecting the optimal image for the boundary coordinate point from multiple sets of images includes:

[0025] Calculate the sum of the brightness gradient values ​​of the K nearest feature units of the boundary coordinate point in all images, and select the image with the largest sum of brightness gradient values ​​as the optimal image of that point.

[0026] Specifically, the method for fusing all corrected boundary coordinate points includes:

[0027] A weighted average method is used for fusion, where the weight of the boundary coordinate point in each image is determined based on the local brightness gradient value of the point in the image or the brightness gradient value of the feature unit to which it belongs.

[0028] Specifically, the method also outputs a high dynamic range (HDR) fused image, which is generated by fusing the pixel values ​​of each pixel from multiple sets of images, selecting the pixel values ​​with the largest brightness gradient or the brightness values ​​closest to a preset median value.

[0029] This application has the following technical advantages:

[0030] By using multi-exposure image fusion and structured feature analysis, the boundaries of key anatomical structures of the larynx can be automatically and efficiently extracted and optimized, significantly improving the detail representation of highlight and shadow areas in the image. This effectively reduces the workload of multi-image comparison and analysis while ensuring diagnostic accuracy. Attached Figure Description

[0031] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily understood by reading the following detailed description with reference to the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, and the same or corresponding reference numerals denote the same or corresponding parts.

[0032] Figure 1 This is a flowchart of an inspection image processing method based on multi-exposure fusion in an embodiment of this application. Detailed Implementation

[0033] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0034] Laryngeal examination is a common diagnostic method in otolaryngology, using a laryngoscopy to obtain images of the larynx to assess the health of the vocal cords, glottis, and surrounding tissues. However, due to the complex structure of the larynx and limited lighting conditions, especially under cold light sources, the high reflectivity of the laryngeal mucosa and the coexistence of shadows on deep structures make it difficult for a single exposure image to fully capture the details of all key areas. Highlight areas are prone to overexposure, while shadow areas lose detail, greatly hindering clinical diagnosis for physicians. To overcome this limitation, current techniques typically involve acquiring multiple sets of images with different exposures from the same viewpoint for manual comparison or general image fusion. However, this method is inefficient and lacks targeted optimization for anatomical boundaries, failing to meet the clinical demand for efficient and accurate image analysis.

[0035] To address the aforementioned problems, this invention proposes a structured multi-exposure image processing method. Its core lies in introducing triangular mesh feature units and a dual eigenvalue mechanism to achieve automatic association, optimization, and fusion of key boundaries among multiple images, significantly improving the clarity and recognition accuracy of laryngeal anatomical structure boundaries. The following detailed description, in conjunction with the accompanying drawings, illustrates this approach. Figure 1 As shown, the inspection image processing method based on multi-exposure fusion proposed in this application includes the following steps:

[0036] Multiple sets of images with different exposure intensities are acquired from the same inspection perspective;

[0037] The key anatomical structures of the larynx in each group of images are segmented to obtain the initial boundary coordinates of each group of images;

[0038] Within the initial boundary coordinates, a triangular mesh covering the key region is generated using the Deloni triangulation algorithm, where each triangular mesh is defined as a feature cell;

[0039] For each feature unit, calculate its average brightness value and brightness gradient value, thereby generating a brightness feature value containing the average brightness value and brightness gradient value for each feature unit in each group of images.

[0040] Based on the variation of the average brightness of multiple triangular mesh units corresponding to the same anatomical structure in different images, a corresponding correlation relationship of feature units between multiple sets of images is established.

[0041] For each initial boundary coordinate point, based on the brightness gradient value of its neighboring feature units in multiple sets of images with established associations, the optimal image of the boundary coordinate point is selected from multiple sets of images, and the position of the boundary coordinate point is corrected to the corresponding position in the optimal image.

[0042] By fusing all the corrected boundary coordinates, the final reinforced boundary of the key anatomical structures of the larynx is generated.

[0043] In one embodiment of this application, firstly, during the image acquisition stage, multiple sets of raw images with different exposure levels are continuously acquired from the same laryngoscope examination perspective by rapidly adjusting camera exposure parameters or light source intensity. Typically, three to five sets of images are acquired, covering multiple brightness levels from underexposure to overexposure, to ensure that subsequent processing can fully utilize complementary image information under different exposures. Since multiple sets of images are acquired rapidly and continuously from the same perspective, slight displacement or jitter may exist; therefore, image registration is required to ensure coordinate consistency in subsequent processing. In this embodiment, the SIFT operator is used to extract stable feature points in each image, and then the optimal affine transformation matrix is ​​calculated using the RANSAC algorithm to align all images to the reference image coordinate system, thereby eliminating image misalignment caused by slight patient movement or equipment shaking.

[0044] After image registration, a deep learning model based on the U-Net architecture is used to segment key anatomical structures of the larynx. The model's input is a single registered grayscale image or a brightness channel image, and the output is a probability map of each pixel belonging to a specific anatomical structure such as the vocal cords or glottis. To improve the model's sensitivity to low-contrast boundaries and feature selection ability, this embodiment introduces an attention gating mechanism in the skip connections of U-Net. Specifically, at each downsampling stage of the encoder path, its feature map interacts with the feature map of the corresponding layer of the decoder through an attention gating module. By calculating the attention coefficient of the encoder feature map, regions relevant to the current decoding task are emphasized, while irrelevant background information is suppressed.

[0045] To improve the model's ability to adapt to image changes in real clinical environments, this embodiment establishes a high-quality, professionally annotated dataset containing thousands of laryngoscopy images precisely annotated by doctors. The annotated objects include the contours of key structures such as the vocal cords, glottis, and arytenoid cartilage. In addition, various data augmentation strategies are employed during training, including elastic deformation to simulate tissue softness, random rotation and scaling to simulate different examination angles, and brightness and contrast adjustment to simulate varying lighting conditions.

[0046] After obtaining the initial boundary, a feature cell mesh needs to be constructed within the boundary range. This embodiment uses the Deloni triangulation algorithm to systematically generate the triangular mesh. Specifically, feature points are first sampled inside the boundary. The sampling strategy can be adaptively adjusted according to the image content; for example, dense sampling is performed in areas with high gradients and rich details, while sparse sampling is performed in flat areas to better capture structural features. After merging the boundary points with the internal sampled points, Deloni triangulation is performed. This algorithm can generate regular-shaped, nearly equiangular triangular cells, effectively avoiding the computational bias caused by elongated triangles. Each triangular cell is defined as a feature cell for subsequent semi-local feature analysis.

[0047] After obtaining the feature units, the system calculates the average brightness value and brightness gradient value of the feature units using the following methods:

[0048] Extract the V channel values ​​of all pixels within the area covered by each triangular mesh feature unit in the HSV color space, and calculate their arithmetic mean to obtain the average brightness value;

[0049] The Sobel operator is used to calculate the gradient magnitude of all pixels within the region, and the average gradient value is obtained to get the brightness gradient value.

[0050] In this embodiment, for each input image, it is first converted from the original RGB color space to the HSV color space to separate luminance information from color information, avoiding the influence of color differences on luminance and color. For each triangular mesh unit, the values ​​of all pixels covered by it in the V channel are extracted, and their arithmetic mean is calculated, which is the luminance average (AB) of that unit, reflecting the overall brightness of that local area at a specific exposure level. Simultaneously, the Sobel operator is used to calculate the gradient components of each pixel in the x and y directions, thereby obtaining its gradient magnitude. The average of the gradient magnitudes of all pixels within the entire triangular unit is defined as the luminance gradient (BG) of that unit. In moderately exposed images, the texture is clear and the edges are sharp; its BG value will be significantly higher than the value of the same area in overexposed or underexposed images.

[0051] After calculating the double eigenvalues ​​of all feature units in all images, the system begins to perform cross-image feature unit association, including the following methods:

[0052] For any triangular mesh cell in any image, candidate cells are found in another image based on their spatial topological location. The correlation coefficient of the average brightness sequence of the two cells in different exposure images is calculated, and the candidate cell with the largest correlation coefficient is established as the corresponding associated cell.

[0053] In this embodiment, leveraging the characteristic that the average brightness (AB) of the same anatomical structure varies according to a certain pattern under different exposures, for a specific triangular unit in the reference image, the system first initially filters out several candidate units with similar positions in the target image based on its spatial topological location within the entire grid. Subsequently, the system constructs the AB value sequence of the reference unit in all N exposed images, and similarly constructs the AB value sequence of each candidate unit in the N images. The consistency of their brightness variation trends is evaluated by calculating the Pearson correlation coefficient between the reference unit sequence and each candidate unit sequence. Finally, the candidate unit with the highest correlation coefficient is established as the corresponding unit.

[0054] After establishing the association relationships among all feature units, the system begins to correct the initial boundary. It calculates the sum of the brightness gradient values ​​of the K nearest feature units of the boundary coordinate point across all images, and selects the image with the largest sum of brightness gradient values ​​as the optimal image for that point. In this embodiment, for each coordinate point on the boundary obtained from the initial segmentation, the system locates its K nearest triangular feature units. Subsequently, it traverses all N exposure images, finding the associated units corresponding to these K units in each image, and summing their brightness gradient values ​​(BG) to obtain an aggregate index representing the overall sharpness of the image in this local area. The system selects the image with the largest aggregate gradient value, determining it as the best source image presenting the sharpest point at that specific boundary. The coordinates of this boundary point are corrected to the coordinates of the corresponding position extracted from this optimal image.

[0055] Finally, after the above-mentioned selection and correction, all boundary points form a new set of boundary points stitched together from the clearest boundary segments from different source images. Before generating the final enhanced boundary, a weighted average fusion strategy is used to smooth the corrected boundary point set. The weight of each boundary point can be determined by its own gradient value in the optimal image or the gradient value of its corresponding feature unit. Points with higher gradient values ​​are considered to be more reliable in their localization and are given higher weights during the smoothing process, thus ensuring the natural continuity of the boundary while maintaining sharpness. Finally, this smooth and high-precision boundary is superimposed on an HDR background image with good overall visual effect generated by a traditional multi-exposure fusion algorithm, and the output is the final enhanced result image. The enhanced boundary is obtained by fusing the pixel values ​​with the most moderate brightness or the largest gradient from each pixel in the multi-exposure sequence, and is finally superimposed on the fused background with a bright color, outputting an enhanced image that can be directly used clinically.

[0056] Obviously, the embodiments described above are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0057] It should be understood that when the terms "first," "second," etc., are used in the claims, description, and drawings of this application, they are only used to distinguish different objects and not to describe a specific order. The terms "comprising" and "including" used in the description and claims of this application indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

Claims

1. An inspection image processing method based on multi-exposure fusion, characterized in that, Includes the following steps: Multiple sets of images with different exposure intensities are acquired from the same inspection perspective; The key anatomical structures of the larynx in each group of images are segmented to obtain the initial boundary coordinates of each group of images; Within the range of the initial boundary coordinates, a triangular mesh covering the key region is generated using the Deloni triangulation algorithm, wherein each triangular mesh is defined as a feature cell; For each feature unit, calculate its average brightness value and brightness gradient value, thereby generating a brightness feature value containing the average brightness value and brightness gradient value for each feature unit in each group of images. Based on the variation of the average brightness of multiple triangular mesh units corresponding to the same anatomical structure in different images, a corresponding correlation relationship of feature units between multiple sets of images is established. For each initial boundary coordinate point, based on the brightness gradient value of its neighboring feature units in multiple sets of images with established associations, the optimal image of the boundary coordinate point is selected from multiple sets of images, and the position of the boundary coordinate point is corrected to the corresponding position in the optimal image. By fusing all the corrected boundary coordinates, the final reinforced boundary of the key anatomical structures of the larynx is generated.

2. The method according to claim 1, characterized in that, The step of segmenting the key anatomical structures of the larynx in each set of images includes: Before the step of semantic segmentation of key anatomical structures of the larynx in each group of images, image registration is performed on the multiple groups of images with different exposure intensities. The key anatomical structures of the larynx, including the vocal cords, glottis, and arytenoid cartilage, were segmented using a deep learning model based on the U-Net architecture.

3. The method according to claim 2, characterized in that, The deep learning model based on the U-Net architecture incorporates an attention mechanism module during training and is trained using a dataset of laryngoscope images annotated by experts, employing data augmentation techniques such as elastic deformation, random rotation, and brightness adjustment.

4. The method according to claim 1, characterized in that, The method for generating triangular meshes covering key regions using the Deloni triangulation algorithm includes: Uniform or image gradient-based adaptive feature point sampling is performed within the critical anatomical region enclosed by the initial boundary coordinates. All boundary coordinate points and internal sampling points are merged into an input point set, and the Deloni triangulation algorithm is performed on the input point set to generate a triangular mesh.

5. The method according to claim 1, characterized in that, The method for calculating its average brightness and brightness gradient value includes: Extract the V channel values ​​of all pixels within the area covered by each triangular mesh feature unit in the HSV color space, and calculate their arithmetic mean to obtain the average brightness value; The Sobel operator is used to calculate the gradient magnitude of all pixels in the region, and the average gradient value is obtained to obtain the brightness gradient value.

6. The method according to claim 1, characterized in that, The method of establishing the correspondence between feature units in multiple sets of images based on the variation of the average brightness of multiple triangular mesh units corresponding to the same anatomical structure in different images includes: For any triangular mesh cell in any image, candidate cells are found in another image based on their spatial topological location. The correlation coefficient of the average brightness sequence of the two cells in different exposure images is calculated, and the candidate cell with the largest correlation coefficient is established as the corresponding associated cell.

7. The method according to claim 1, characterized in that, The method for selecting the optimal image for the boundary coordinate point from multiple sets of images includes: Calculate the sum of the brightness gradient values ​​of the K nearest feature units of the boundary coordinate point in all images, and select the image with the largest sum of brightness gradient values ​​as the optimal image of that point.

8. The method according to claim 1, characterized in that, The method for fusing all corrected boundary coordinate points includes: A weighted average method is used for fusion, where the weight of the boundary coordinate point in each image is determined based on the local brightness gradient value of the point in the image or the brightness gradient value of the feature unit to which it belongs.

9. The method according to claim 1, characterized in that, The method also outputs a high dynamic range (HDR) fused image, which is generated by fusing the pixel values ​​of each pixel from multiple sets of images, selecting the pixel values ​​with the largest brightness gradient or the brightness values ​​closest to a preset median value.

Images