A lesion image processing method and system based on multi-modal fusion

Abstract

The present application belongs to the technical field of medical image processing, and provides a lesion image processing method and system based on multi-modal fusion, which comprises: acquiring an endoscope image and an ultrasound image of the same anatomical position, extracting an optical highlight area in the endoscope image based on a brightness and gradient double threshold, extracting a strong echo area and a corresponding rear acoustic shadow area in the ultrasound image and merging them into an acoustic suspected artifact area, uniformly mapping the two types of areas to a three-dimensional physical space coordinate system to calculate the spatial coincidence degree, identifying and accurately positioning the micro-bubble artifact area according to a judgment threshold, performing cross-modal collaborative repair of the two types of images by using normal tissue pixels around the artifact area, eliminating mirror highlight, strong echo and acoustic shadow interference, and finally performing spatial registration and fusion on the repaired image to output a lesion fusion image without artifacts and with complete features, thereby improving the reliability of endoscopic ultrasound images.

Description

Technical Field

[0001] This invention belongs to the field of Y technology, specifically a method and system for lesion image processing based on multimodal fusion; Background Technology Endoscopic ultrasound technology combines visual observation of the endoscopic surface with deep ultrasound tomography, requiring registration, segmentation, and fusion of simultaneously acquired endoscopic optical images and ultrasound images. Clinical endoscopic ultrasound examinations require the injection of water into the digestive tract to form an acoustic coupling pathway. The injection of water and the agitation of the probe can easily leave microbubbles on the surface of lesions, in mucosal folds, or in tiny depressions.

[0002] Microbubbles attached to the surface of lesions can create complex interference in both modes: in endoscopic images, they produce strong specular highlights that obscure the microvascular and glandular textures on the lesion surface; in ultrasound images, they form strong echoic patches that create acoustic shadows behind the lesion, obscuring the depth and basal boundary information of the lesion.

[0003] Existing processing methods are mostly designed for independent denoising or feature extraction of a single mode, and cannot cope with the dual acoustic-optical artifacts caused by optical highlights and strong acoustic echoes with sound shadows simultaneously caused by microbubbles in the same spatial location, resulting in cross-modal feature conflicts.

[0004] Traditional segmentation and fusion algorithms struggle to distinguish between bimodal artifacts and real pathological tissues, often misclassifying artifacts as calcification, necrosis, or high-density nodules. This leads to errors in lesion boundary segmentation, visual misalignment and feature tearing in fused images, and ultimately, misleads clinical diagnosis.

[0005] Overcoming the acoustic-optical artifacts caused by microbubbles in water injection coupling scenarios, avoiding the limitations of single-modal processing and misjudgments in dual-modal fusion, and achieving high-precision lesion image processing and fusion are urgent technical problems to be solved.

[0006] Therefore, the present invention provides a method and system for lesion image processing based on multimodal fusion. Summary of the Invention

[0007] In order to overcome the shortcomings of the prior art and solve at least one technical problem raised in the background art; The technical solution adopted by this invention to solve its technical problem is: In one aspect, the present invention provides a lesion image processing method based on multimodal fusion, comprising: S1: Acquire endoscopic and ultrasound images of the same anatomical location; extract optical highlight regions in the endoscopic images that satisfy the first brightness threshold and the first gradient threshold; S2: Extract the strong echo region that meets the second brightness threshold from the ultrasound image, and the acoustic shadow region located behind the sound beam propagation of the strong echo region. Merge them into an acoustic suspected artifact region. Map the optical highlight region and the acoustic suspected artifact region to the same three-dimensional physical space coordinate system and calculate the spatial overlap. S3: The overlapping regions with a spatial overlap degree greater than the preset artifact joint judgment threshold are judged as microbubble artifact regions. S4: Using the normal pixels around the microbubble artifact region, perform cross-modal collaborative repair on endoscopic and ultrasound images respectively, and then fuse the repaired dual-modal images.

[0008] As a further improvement of the present invention, the specific process of acquiring endoscopic images and ultrasound images of the same anatomical location is as follows: An endoscopic ultrasound device that integrates an optical camera and an ultrasound transducer at the probe tip can simultaneously scan and image the same lesion area in the digestive tract under water-coupled conditions. By using a hardware timestamp synchronization mechanism, a frame of endoscopic image and a frame of ultrasound image with the same time sequence identifier are selected to achieve paired acquisition of endoscopic and ultrasound images.

[0009] As a further improvement of the present invention, the specific process of extracting the optical highlight region that satisfies the first brightness threshold and the first gradient threshold in the endoscopic image is as follows: The color endoscopic image is converted to grayscale space or the V brightness channel of HSV color space for processing. A first brightness threshold is set according to the upper limit of the dynamic range of the endoscopic image sensor. The value is 240 to 250 in the 8-bit grayscale image, or the top 2% of grayscale values ​​in the global brightness histogram are taken. Pixels with grayscale values ​​greater than or equal to the first brightness threshold are selected to form a bright candidate mask. An edge detection operator is used to calculate the grayscale gradient amplitude of the entire image. The first gradient threshold is set to 3 to 5 times the average background gradient amplitude of normal mucosal tissue. The bright candidate mask and the global grayscale gradient amplitude map are logically ANDed. Each pixel and the edge pixels of the 8 connected domain are judged. When the pixel grayscale value is greater than or equal to the first brightness threshold and the preset maximum gradient amplitude of the neighborhood is greater than or equal to the first brightness threshold, it is judged as an optical highlight area and extracted.

[0010] As a further improvement of the present invention, the specific process of extracting the strong echo region that satisfies the second brightness threshold in the ultrasound image is as follows: The acquired ultrasound images are traversed, and the second brightness threshold is set to 90% to 95% of the global maximum gray value of the ultrasound image. Connected pixels with gray values ​​greater than or equal to the second brightness threshold are extracted and marked as the strong echo region.

[0011] As a further improvement of the present invention, the specific process for extracting the sound shadow region located behind the sound beam propagation in the strong echo region is as follows: Obtain the sound beam propagation direction vector corresponding to the centroid of the strong echo region. Perform a pixel-by-pixel search in the far field of the strong echo region along the sound beam propagation direction vector. Set a low echo judgment threshold and extract the strip-shaped regions with echo intensity lower than the low echo judgment threshold along the sound beam propagation direction and mark them as sound shadow regions.

[0012] As a further improvement of the present invention, the specific process for obtaining the acoustic suspected artifact region is as follows: The spatial locations of the strong echo region and the acoustic shadow region within the ultrasound image are determined. The strong echo region and the acoustic shadow region are spliced ​​and fused together by morphological closing operation or logical OR operation to form a continuous and complete whole region, which is denoted as the acoustic suspected artifact region.

[0013] As a further improvement of the present invention, the specific process for calculating the spatial overlap is as follows: By utilizing the hand-eye calibration parameters of the endoscope-ultrasound probe, which are pre-calibrated at the factory, using the ultrasonic endoscope equipment that integrates an endoscope optical camera and an ultrasonic transducer, a unified three-dimensional physical space coordinate system is established. Based on the internal parameters of the endoscope's optical camera and the pixel-to-physical distance conversion coefficient of the ultrasound image, the optical highlight area is back-projected into the three-dimensional physical space coordinate system, and the acoustic suspected artifact area is mapped into the three-dimensional physical space coordinate system. In a three-dimensional physical space coordinate system, the spatial intersection-union ratio (IoU) between the mapped optical specular region and the acoustic suspected artifact region is calculated, and the resulting ratio is the spatial overlap.

[0014] As a further improvement of the present invention, the specific process for determining the microbubble artifact region is as follows: The calculated spatial overlap is compared with the preset artifact joint judgment threshold: If the spatial overlap is not lower than the preset artifact joint judgment threshold, the optical highlight area and the acoustic suspected artifact area are spatially merged in a unified three-dimensional physical space coordinate system, and the merged overall area is marked as the finally determined microbubble artifact area.

[0015] As a further improvement of the present invention, the specific process of cross-modal collaborative restoration of endoscopic images and ultrasound images is as follows: In the endoscopic image, the microbubble artifact region is used as the repair target mask. A ring-shaped region of a preset width outside the repair target mask boundary is extracted as the normal optical reference area. Using the image inpainting algorithm based on partial differential equation (PDE) or the image patch-based inpainting technique based on texture synthesis, the damaged highlight pixels are iteratively filled from the edge of the repair target mask to the center, guided by the pixel color values ​​and gradient directions in the normal optical reference area, to generate the repaired endoscopic image. In the ultrasound image, the microbubble artifact region is used as the repair target mask. The normal ultrasound echo tissues that are not obscured by the acoustic shadow are extracted from the two sides of the repair target mask and used as normal acoustic reference areas. Along the horizontal direction of the ultrasound image, the echo intensity distribution at the corresponding depth in the normal acoustic reference area is used to perform weighted average interpolation or bicubic interpolation smoothing on the strong echo pixels and low echo pixels in the repair target mask to generate the repaired ultrasound image.

[0016] On the other hand, the present invention provides a lesion image processing system based on multimodal fusion, comprising: Image acquisition and optical region extraction module: acquires endoscopic and ultrasound images of the same anatomical location; extracts optical highlight regions in the endoscopic images that satisfy a first brightness threshold and a first gradient threshold; Ultrasonic artifact region extraction and spatial registration module: Extract strong echo regions that meet the second brightness threshold from the ultrasound image, as well as acoustic shadow regions located behind the sound beam propagation of the strong echo regions, and merge them into acoustic suspected artifact regions. Map the optical highlight regions and acoustic suspected artifact regions to the same three-dimensional physical space coordinate system and calculate the spatial overlap. Microbubble artifact detection module: Identifies overlapping regions with spatial overlap greater than a preset artifact joint detection threshold as microbubble artifact regions. Cross-modal image restoration and fusion module: Utilizes normal pixels around the microbubble artifact region to perform cross-modal collaborative restoration of endoscopic and ultrasound images respectively, and then fuses the restored dual-modal images.

[0017] The beneficial effects of this invention are as follows: 1. This invention maps the optical highlight area of ​​the endoscope image and the acoustic suspected artifact area of ​​the ultrasound image to the same three-dimensional physical space coordinate system, calculates the spatial overlap and combines it with a preset threshold to accurately locate the microbubble artifact area. This effectively avoids the problem of existing technologies misjudging microbubble artifacts as real pathological tissues such as calcification points and deep necrosis areas, and provides accurate imaging evidence for clinical diagnosis.

[0018] 2. Completely eliminate dual-modal artifact interference and restore the true features of lesions: Unlike the limitations of existing single-modal denoising methods, this invention adopts a cross-modal collaborative repair strategy, which utilizes the normal tissue pixels around the artifact area to eliminate the specular highlights in the endoscopic image, the strong echoes and acoustic interference in the ultrasound image, and completely restore the surface information of the lesion, such as the microvessels and glandular textures, as well as the tomographic structure and depth boundaries of the deep tissue, thereby improving the integrity and clarity of the image. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings; Figure 1 This is a flowchart of the steps of a lesion image processing method based on multimodal fusion according to the present invention; Figure 2 This is a system module diagram of a lesion image processing system based on multimodal fusion according to the present invention. Detailed Implementation

[0020] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments; Example 1 like Figure 1 As shown in the embodiment of the present invention, a lesion image processing method and system based on multimodal fusion includes: S1: Acquire endoscopic and ultrasound images of the same anatomical location; extract optical highlight regions in the endoscopic images that satisfy a first brightness threshold and a first gradient threshold; In practice, an endoscopic ultrasound (EUS) device that integrates both an optical camera and an ultrasound transducer at the probe tip is used to simultaneously scan and image the same lesion area in the digestive tract under water-coupled conditions. Through the device's hardware timestamp synchronization mechanism, an endoscopic image and an ultrasound image with the same time sequence identifier are selected to ensure that the two images correspond to the same time and the same scanning field of view. Since the optical camera and ultrasound transducer have a fixed and known physical spatial installation offset at the probe tip, the images acquired by the two at the same time can respectively characterize the surface appearance information and deep tissue tomographic information of the lesion at the same anatomical location, thereby realizing the paired acquisition of endoscopic images and ultrasound images at the same anatomical location. Extract the optical highlight region that satisfies the first brightness threshold and the first gradient threshold from the endoscopic image; Because microbubbles are spherical liquid-gas interfaces, their surfaces experience strong specular reflection under direct illumination from a strong endoscopic light source, leading to severe overexposure of pixels in that area. Simultaneously, the highlighted areas of microbubbles exhibit steep edges with abrupt changes in grayscale compared to the surrounding normal pink mucosal tissue, unlike the naturally smooth color transitions of normal tissue. Therefore, this invention employs both brightness and gradient constraints to precisely lock the optical imaging features of microbubbles. The specific extraction steps are as follows: The acquired color endoscopic image is converted to grayscale space, or the V brightness channel of the HSV color space is extracted and processed. The pixels in the mirror reflection area formed by microbubbles are generally in a grayscale saturation state, and a first brightness threshold is set accordingly. The first brightness threshold is set based on the upper limit of the dynamic range of the image sensor of the endoscopic optical camera. In an 8-bit grayscale image (pixel value range 0-255), the first brightness threshold can be a fixed empirical value (such as 245) in the range of 240 to 250, or an adaptive threshold method can be used to select the pixel values ​​of the top 2% of grayscale values ​​in the global brightness histogram of the current endoscopic image. The entire endoscopic image is traversed, and all pixels with grayscale values ​​greater than or equal to the first brightness threshold are selected to generate a high-brightness candidate mask. Relying solely on a single first brightness threshold for screening can easily misidentify white mucus on the lesion surface, highly reflective smooth polyps, and pale ischemic tissue as microbubbles. To eliminate these interfering targets, gradient features are introduced for secondary verification. Edge detection operators such as Sobel and Laplacian are used to calculate the spatial gray-level gradient magnitude of the entire endoscopic image pixel by pixel, forming a global gray-level gradient magnitude map, and a first gradient threshold is set accordingly. In the endoscopic image, a normal mucosal tissue background reference area is selected, and the average gradient amplitude of all pixels in the area is calculated to obtain the average background gradient amplitude of the normal mucosal tissue. The first gradient threshold is used to distinguish between the smooth transition edges of normal tissue and the sharp abrupt edges of microbubble artifacts: the grayscale color of normal mucosal tissue is distributed in a gentle gradient, and the local gradient amplitude is generally low; the reflected light intensity at the edge of microbubble shows a cliff-like attenuation within only a few pixels, and the local gradient amplitude is significantly high. Based on this characteristic, the first gradient threshold is set to 3 to 5 times the average background gradient amplitude of normal mucosal tissue. Perform a spatial logical AND operation between the aforementioned candidate highlight mask and the global grayscale gradient magnitude map; the specific determination rule is as follows: For each pixel in the endoscope image and the edge pixels of its 8 connected domains, when the pixel's own gray value is greater than or equal to the first brightness threshold, and the maximum gradient amplitude within the preset neighborhood of the pixel is greater than or equal to the first gradient threshold, the pixel region is determined to be a specular reflection artifact region formed by microbubbles in the optical mode, thereby completing the accurate extraction of the optical highlight region. S2: Extract the strong echo region that meets the second brightness threshold from the ultrasound image, and the acoustic shadow region located behind the sound beam propagation of the strong echo region, and combine them into an acoustic suspected artifact region; map the optical highlight region and the acoustic suspected artifact region to the same spatial coordinate system, and calculate the spatial overlap between the two. First, strong echo regions that meet the second brightness threshold are extracted from the ultrasound image, and acoustic shadow regions located behind the sound beam propagation of the strong echo regions are combined into acoustic suspected artifact regions. Because of the significant difference in acoustic impedance between the gas inside the microbubbles and the surrounding water medium or digestive tract tissue, ultrasound waves undergo almost total reflection when they come into contact with the surface of the microbubbles. This physical phenomenon manifests in the ultrasound image as two closely connected features: an extremely bright patch at the front and a dark band behind it. The specific extraction steps are as follows: The acquired ultrasound images (B-mode ultrasound grayscale images) are traversed, and a second brightness threshold is set. The physical significance of the second brightness threshold is to distinguish between the extremely high echoes caused by total internal reflection of sound waves due to microbubbles and the high echoes of normal submucosal or muscularis propria. Typically, the echo intensity of microbubbles is much higher than that of normal tissue. Therefore, the second brightness threshold is set as the upper limit range of the global grayscale histogram of the ultrasound image, specifically 90% to 95% of the maximum grayscale value of the ultrasound image. Connected pixels in the ultrasound image with grayscale values ​​greater than or equal to the second brightness threshold are extracted and marked as the strong echo regions. Because ultrasonic energy is largely reflected on the surface of microbubbles, the sound wave energy that penetrates the microbubbles and enters deep tissues is drastically attenuated, thus forming an anechoic or hypoechoic black stripe, i.e., an acoustic shadow, directly behind the microbubbles. Based on the scanning geometry of the ultrasonic transducer (such as radial in a ring array scan or parallel in a linear array scan), the propagation direction vector of the sound beam corresponding to the centroid of the strong echo region is first obtained. Along this propagation direction vector, a pixel-by-pixel search is performed in the far field of the strong echo region (i.e., the side with deeper ultrasonic detection depth). At the same time, a low echo threshold is set (e.g., grayscale value less than 15 or close to the background noise level), and the strip-shaped region with echo intensity lower than the low echo threshold along the propagation direction of the sound beam is extracted and marked as the acoustic shadow region. The spatial locations of the strong echo region (artifact head) and the acoustic shadow region (artifact tail) in the ultrasound image are determined respectively, and region fusion processing is performed based on their spatial distribution relationship. The fusion method adopts morphological closing operation to achieve structural regularity, or directly adopts logical OR operation to achieve region splicing, and merges the two regions into a continuous and complete whole region in space, which is denoted as the acoustic suspected artifact region. The optical highlight region and the acoustic suspected artifact region are mapped to the same spatial coordinate system, and their spatial overlap is calculated. Because the endoscopic optical camera and the ultrasonic transducer are physically mounted at different positions on the probe tip, the endoscopic image and the ultrasonic image are in different sensor coordinate systems. The specific mapping and calculation steps are as follows: Using the endoscope-ultrasound probe hand-eye calibration parameters (i.e., external parameter matrix including rotation matrix and translation vector) pre-calibrated at the factory of the ultrasonic endoscope device integrating an endoscope optical camera and an ultrasonic transducer, a unified three-dimensional physical space coordinate system (i.e., the same spatial coordinate system) is established. For example, the three-dimensional physical space coordinate system is constructed with the center of the ultrasonic transducer array as the origin, the central axis of the ultrasonic beam as the Z-axis, the horizontal direction of the array perpendicular to the beam as the X-axis, and the vertical direction as the Y-axis; the two-dimensional image coordinate system where the endoscope optical camera is located is transformed to this unified three-dimensional physical space coordinate system through the hand-eye calibration parameters. Based on the internal parameters of the endoscope optical camera (including focal length and distortion coefficient) and the pixel-to-physical distance conversion coefficient of the ultrasound image (exemplarily 0.1 mm / pixel, a factory preset parameter), the two-dimensional optical highlight region extracted in step S1 is back-projected onto the mucosal surface position in the three-dimensional physical space coordinate system; at the same time, the two-dimensional acoustic suspected artifact region extracted in this step (especially the strong echo region at its front end) is synchronously mapped onto the three-dimensional physical space coordinate system; In the three-dimensional physical space coordinate system, the spatial intersection-union ratio (IoU) of the mapped optical specular region and the acoustic suspected artifact region (front-end strong echo part) is calculated; specifically, the ratio of the intersection volume or projected intersection area of ​​the two in three-dimensional space to the total volume or total projected area of ​​the two is calculated, and the obtained ratio is the spatial overlap. For example, using the two-dimensional projection plane of the mucosal surface in three-dimensional space as the calculation benchmark, the pixel set A occupied by the optical highlight region and the pixel set B occupied by the acoustic strong echo region are statistically analyzed. The ratio of the number of intersecting pixels to the number of merging pixels between the two is calculated using the formula IoU=|A∩B| / |A∪B|. This ratio is the spatial overlap. This spatial overlap is used to determine whether optical highlights and strong acoustic echoes originate from the same entity in physical space (i.e., the same microbubble attached to the mucosal surface). S3: When the spatial overlap is greater than the preset artifact joint determination threshold, the overlapping area is determined to be a microbubble artifact area. After completing the cross-modal space mapping in step S2, this step aims to accurately eliminate spurious lesions (i.e., microbubbles) through cross-validation of bimodal information, avoiding confusion with real pathological tissues (such as submucosal calcifications or necrotic nodules); the specific judgment logic and implementation steps are as follows: In real clinical scenarios, although calcifications or high-density nodules inside lesions may appear as strong echoes (even with posterior acoustic shadowing) in ultrasound images, they do not produce specular highlights on the mucosal surface in endoscopic images because they are located in the submucosa or deeper. Instead, only when microbubbles generated by water injection into the digestive tract adhere to the mucosal surface will they simultaneously produce the optically bright areas in endoscopic images and the acoustically suspected artifact areas (i.e., strong echoes and acoustic shadowing) in the same physical location in ultrasound images. Therefore, by calculating the spatial overlap of the two in the same spatial coordinate system, the specific identification of microbubbles can be achieved. Obtain the spatial overlap (Intersection over Union (IoU) value) calculated in step S2; here, set a preset artifact joint judgment threshold; the physical meaning of the artifact joint judgment threshold is: to tolerate slight spatial registration deviations caused by minute vibrations, respiratory movements or calibration errors between the endoscope and the ultrasound probe, while ensuring that only acoustic-optical dual features that are highly overlapped in space are judged as artifacts. Specifically, considering that the highlight area of ​​microbubbles in optical images is usually slightly smaller than their strong echo cross-sectional area in ultrasound images (limited by the light source illumination angle and the width of the ultrasound beam), the preset artifact joint judgment threshold is usually set between 0.5 and 0.75 (i.e., 50% to 75% overlap rate); for example, an empirical value of 0.6 can be taken. The calculated spatial overlap is compared with the preset artifact joint determination threshold: If the spatial overlap is not lower than the preset artifact joint judgment threshold, it indicates that the optical highlight area and the acoustic suspected artifact area correspond to the composite artifact generated by the same microbubble on the mucosal surface; at this time, the optical highlight area and the acoustic suspected artifact area are spatially merged in a unified three-dimensional physical space coordinate system, and the merged overall area is marked as the finally determined microbubble artifact area. If the spatial overlap is less than or equal to the preset artifact joint judgment threshold, it indicates that there is only a single modality of abnormality at that location (e.g., only strong ultrasound echo without surface highlight, suggesting a possible real submucosal calcification; or only surface highlight without strong ultrasound echo, suggesting a possible normal reflective mucus). The original image features of these areas will be preserved and they will not be judged as artifacts, thereby protecting the real pathological information from being mistakenly deleted to the greatest extent. S4: Using the normal pixels around the microbubble artifact region, perform cross-modal collaborative repair on the endoscopic image and the ultrasound image respectively, and then fuse the repaired dual-modal images.

[0021] After accurately locating the microbubble artifact region in step S3, the strong optical highlights and acoustic total reflection (and acoustic shadows) generated by the microbubbles essentially obscure the original true lesion surface texture and deep tomographic structure of the region. Directly fusing images with artifacts would lead to serious visual distortions and diagnostic misrepresentations. Therefore, this invention adopts a cross-modal collaborative repair strategy based on the characteristics of surrounding normal tissue, specifically implemented as follows: In the endoscopic image, the microbubble artifact region (i.e., the original optical highlight region) determined in step S3 is used as the repair target mask. A ring-shaped region with a preset width (e.g., 5 to 15 pixels wide) around the boundary of the repair target mask is extracted as a normal optical reference region. This normal optical reference region contains the true mucosal color and microvascular texture features of the lesion surface. Using an image inpainting algorithm based on partial differential equations (PDEs) (such as the Navier-Stokes equation or the FastMarching Method) or a texture synthesis-based image inpainting technique (Patch-based Inpainting), guided by the pixel color values ​​and gradient directions within the normal optical reference region, the damaged highlight pixels are iteratively filled from the edge of the repair target mask towards the center. Through the above steps, the overexposed area masked by the microbubbles will be replaced by the surrounding smooth and natural mucosal texture, generating a repaired endoscopic image and eliminating glaring specular reflection interference. In the ultrasound image, the microbubble artifact region (including the strong echo region at the front and the acoustic shadow region behind it) determined in step S3 is used as the repair target mask. Since the ultrasound image reflects the physical density and acoustic impedance cross-section of the tissue, its repair logic differs from the optical texture of a pure surface. Normal ultrasound echo tissue adjacent to the repair target mask and not obscured by the acoustic shadow is extracted as a normal acoustic reference area. Considering that the digestive tract wall structure (such as the mucosa, submucosa, and muscularis propria) has layered lateral continuity in local space, this invention adopts a self-gradient method based on lateral gradients. An adaptive structural interpolation algorithm is used. Specifically, along the transverse direction of the ultrasound image (perpendicular to the direction of sound beam propagation), the echo intensity distribution at the corresponding depth within the normal acoustic reference area is used to perform weighted average interpolation or bicubic interpolation smoothing on the strong echo pixels (bubble surface) and low echo pixels (back acoustic shadow) within the mask of the repair target. Through the above steps, the deep tube wall structure that was blocked by the total internal reflection of the microbubble is reconnected in the transverse direction, generating a repaired ultrasound image that restores the approximate tomographic morphology of the lesion in the acoustic shadow area, avoiding misjudgment as deep necrosis or perforation. After completing the above-mentioned cross-modal collaborative repair, both the repaired endoscopic image and the repaired ultrasound image have removed the acoustic-optical dual artifacts caused by the same microbubble, and retain the true lesion features.

[0022] Finally, using the unified three-dimensional physical space coordinate system established in step S2 (i.e., the same spatial coordinate system), as well as the endoscopic-ultrasound hand-eye calibration external parameter matrix and camera internal parameters, the repaired ultrasound image (usually a two-dimensional tomographic slice or three-dimensional reconstructed volume data) is registered and superimposed onto the three-dimensional physical space position corresponding to the repaired endoscopic image; Alpha blending or pseudo-color mapping technology can be used to achieve a smooth transition and joint display of the optical texture of the lesion surface and the internal acoustic structure, and finally output a high-precision, artifact-free lesion fusion image for doctors to diagnose.

[0023] Example 2 like Figure 2 As shown, based on the specific implementation process of Embodiment 1, the present invention provides a lesion image processing system based on multimodal fusion, comprising: Image acquisition and optical region extraction module: acquires endoscopic and ultrasound images of the same anatomical location; extracts optical highlight regions in the endoscopic images that satisfy a first brightness threshold and a first gradient threshold; Ultrasonic artifact region extraction and spatial registration module: Extract strong echo regions that meet the second brightness threshold from the ultrasound image, as well as acoustic shadow regions located behind the sound beam propagation of the strong echo regions, and merge them into acoustic suspected artifact regions. Map the optical highlight regions and acoustic suspected artifact regions to the same three-dimensional physical space coordinate system and calculate the spatial overlap. Microbubble artifact detection module: Identifies overlapping regions with spatial overlap greater than a preset artifact joint detection threshold as microbubble artifact regions. Cross-modal image restoration and fusion module: Utilizes normal pixels around the microbubble artifact region to perform cross-modal collaborative restoration of endoscopic and ultrasound images respectively, and then fuses the restored dual-modal images.

[0024] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the present invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A lesion image processing method based on multi-modal fusion, characterized in that: include: S1: Acquire endoscopic and ultrasound images of the same anatomical location; extract optical highlight regions in the endoscopic images that satisfy the first brightness threshold and the first gradient threshold; S2: Extract the strong echo region that meets the second brightness threshold from the ultrasound image, and the acoustic shadow region located behind the sound beam propagation of the strong echo region. Merge them into an acoustic suspected artifact region. Map the optical highlight region and the acoustic suspected artifact region to the same three-dimensional physical space coordinate system and calculate the spatial overlap. S3: The overlapping regions with a spatial overlap degree greater than the preset artifact joint judgment threshold are judged as microbubble artifact regions. S4: Using the normal pixels around the microbubble artifact region, perform cross-modal collaborative repair on endoscopic and ultrasound images respectively, and then fuse the repaired dual-modal images.

2. The lesion image processing method based on multi-modal fusion according to claim 1, characterized in that, The specific process for obtaining endoscopic and ultrasound images of the same anatomical location is as follows: An endoscopic ultrasound device that integrates an optical camera and an ultrasound transducer at the probe tip can simultaneously scan and image the same lesion area in the digestive tract under water-coupled conditions. By using a hardware timestamp synchronization mechanism, a frame of endoscopic image and a frame of ultrasound image with the same time sequence identifier are selected to achieve paired acquisition of endoscopic and ultrasound images.

3. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process of extracting the optical highlight region that satisfies the first brightness threshold and the first gradient threshold in the endoscopic image is as follows: The color endoscopic image is converted to grayscale space or the V brightness channel of HSV color space for processing. A first brightness threshold is set according to the upper limit of the dynamic range of the endoscopic image sensor. The value is 240 to 250 in the 8-bit grayscale image, or the top 2% of grayscale values ​​in the global brightness histogram are taken. Pixels with grayscale values ​​greater than or equal to the first brightness threshold are selected to form a bright candidate mask. An edge detection operator is used to calculate the grayscale gradient amplitude of the entire image. The first gradient threshold is set to 3 to 5 times the average background gradient amplitude of normal mucosal tissue. The bright candidate mask and the global grayscale gradient amplitude map are logically ANDed. Each pixel and the edge pixels of the 8 connected domain are judged. When the pixel grayscale value is greater than or equal to the first brightness threshold and the preset maximum gradient amplitude of the neighborhood is greater than or equal to the first brightness threshold, it is judged as an optical highlight area and extracted.

4. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process for extracting the strong echo region that meets the second brightness threshold in the ultrasound image is as follows: The acquired ultrasound images are traversed, and the second brightness threshold is set to 90% to 95% of the global maximum gray value of the ultrasound image. Connected pixels with gray values ​​greater than or equal to the second brightness threshold are extracted and marked as the strong echo region.

5. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process for extracting the sound shadow region located behind the sound beam propagation in the strong echo region is as follows: Obtain the sound beam propagation direction vector corresponding to the centroid of the strong echo region. Perform a pixel-by-pixel search in the far field of the strong echo region along the sound beam propagation direction vector. Set a low echo judgment threshold and extract the strip-shaped regions with echo intensity lower than the low echo judgment threshold along the sound beam propagation direction and mark them as sound shadow regions.

6. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process for obtaining the suspected acoustic artifact region is as follows: The spatial locations of the strong echo region and the acoustic shadow region within the ultrasound image are determined. The strong echo region and the acoustic shadow region are spliced ​​and fused together by morphological closing operation or logical OR operation to form a continuous and complete whole region, which is denoted as the acoustic suspected artifact region.

7. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process for calculating the spatial overlap is as follows: By utilizing the hand-eye calibration parameters of the endoscope-ultrasound probe, which are pre-calibrated at the factory, using the ultrasonic endoscope equipment that integrates an endoscope optical camera and an ultrasonic transducer, a unified three-dimensional physical space coordinate system is established. Based on the internal parameters of the endoscope's optical camera and the pixel-to-physical distance conversion coefficient of the ultrasound image, the optical highlight area is back-projected into the three-dimensional physical space coordinate system, and the acoustic suspected artifact area is mapped into the three-dimensional physical space coordinate system. In a three-dimensional physical space coordinate system, the spatial intersection-union ratio (IoU) between the mapped optical specular region and the acoustic suspected artifact region is calculated, and the resulting ratio is the spatial overlap.

8. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process for determining the microbubble artifact region is as follows: The calculated spatial overlap is compared with the preset artifact joint judgment threshold: If the spatial overlap is not lower than the preset artifact joint judgment threshold, the optical highlight area and the acoustic suspected artifact area are spatially merged in a unified three-dimensional physical space coordinate system, and the merged overall area is marked as the finally determined microbubble artifact area.

9. The lesion image processing method based on multimodal fusion according to claim 1, characterized in that, The specific process of cross-modal collaborative restoration of endoscopic and ultrasound images is as follows: In the endoscopic image, the microbubble artifact region is used as the repair target mask. A ring-shaped region of a preset width outside the repair target mask boundary is extracted as the normal optical reference area. Using the image inpainting algorithm based on partial differential equation (PDE) or the image patch-based inpainting technique based on texture synthesis, the damaged highlight pixels are iteratively filled from the edge of the repair target mask to the center, guided by the pixel color values ​​and gradient directions in the normal optical reference area, to generate the repaired endoscopic image. In the ultrasound image, the microbubble artifact region is used as the repair target mask. The normal ultrasound echo tissues that are not obscured by the acoustic shadow are extracted from the two sides of the repair target mask and used as normal acoustic reference areas. Along the horizontal direction of the ultrasound image, the echo intensity distribution at the corresponding depth in the normal acoustic reference area is used to perform weighted average interpolation or bicubic interpolation smoothing on the strong echo pixels and low echo pixels in the repair target mask to generate the repaired ultrasound image.

10. A lesion image processing system based on multimodal fusion, used to perform the method described in any one of claims 1-9, characterized in that: include: Image acquisition and optical region extraction module: acquires endoscopic and ultrasound images of the same anatomical location; Extract the optical highlight region that satisfies the first brightness threshold and the first gradient threshold from the endoscopic image; Ultrasonic artifact region extraction and spatial registration module: Extract strong echo regions that meet the second brightness threshold from the ultrasound image, as well as acoustic shadow regions located behind the sound beam propagation of the strong echo regions, and merge them into acoustic suspected artifact regions. Map the optical highlight regions and acoustic suspected artifact regions to the same three-dimensional physical space coordinate system and calculate the spatial overlap. Microbubble artifact detection module: Identifies overlapping regions with spatial overlap greater than a preset artifact joint detection threshold as microbubble artifact regions. Cross-modal image restoration and fusion module: Utilizes normal pixels around the microbubble artifact region to perform cross-modal collaborative restoration of endoscopic and ultrasound images respectively, and then fuses the restored dual-modal images.

Images