An optimization method, device and terminal equipment for multi-modal image display

Abstract

The application is suitable for the technical field of image processing, and provides a multi-modal image display optimization method and device and terminal equipment, which comprises the following steps: obtaining an OCT image and a autofluorescence image of a target blood vessel section, the OCT image being an image obtained through optical coherence tomography, and the autofluorescence image being an image obtained through autofluorescence of a specific plaque component stimulated by near-infrared laser, fusing the OCT image and the autofluorescence image to obtain a target image, each pixel point in the target image being determined according to each pixel point of the OCT image, and the value of each pixel point in the target image being determined according to the autofluorescence intensity coefficient of a pixel point of the autofluorescence image. The application can improve the accuracy of identifying an atherosclerotic plaque in the OCT image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and in particular relates to an optimization method, apparatus, terminal device and computer-readable storage medium for multimodal image display. Background Technology

[0002] Atherosclerosis, also known as arteriosclerosis, refers to the deposition of lipids and inflammatory substances on the inner walls of arteries, forming plaques that resemble millet porridge in appearance. This narrows and hardens the arteries, obstructing blood flow. Atherosclerosis can affect arteries throughout the body, such as the coronary arteries, renal arteries, and cerebral arteries, as well as organs such as the heart, kidneys, and brain. Because mild cases of atherosclerosis generally do not present symptoms, when the condition progresses, especially when plaques rupture and secondary thrombosis occurs, blood flow can be suddenly interrupted, potentially leading to refractory hypertension, angina pectoris, cerebral ischemia, and in severe cases, even sudden death. Therefore, accurately identifying the progression and severity of atherosclerosis is crucial for disease diagnosis and treatment.

[0003] Currently, atherosclerotic plaques are typically examined using imaging or angiography. Among these, Optical Coherence Tomography (OCT) is a novel biomedical imaging technique developed in recent years. It uses low-coherence light to acquire two-dimensional or three-dimensional images of biological tissue with micron-level resolution, which can be used to differentiate atherosclerosis and the resulting vascular blockage or rupture. OCT has a penetration depth of approximately 2 mm, allowing analysis of most subendothelial atherosclerotic plaques and vascular imaging. However, for some larger atherosclerotic plaques and the walls of large arteries, the information obtained from OCT imaging is limited, affecting the diagnosis of atherosclerotic plaques in OCT images. Summary of the Invention

[0004] This application provides an optimization method, apparatus, and terminal device for multimodal image display, which can solve the problem that the existing OCT technology has poor tissue penetration, resulting in the inability to identify the full picture of the patch based on the OCT image.

[0005] In a first aspect, embodiments of this application provide an optimization method for multimodal image display, comprising:

[0006] Obtain OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser.

[0007] The target image is obtained by fusing the OCT image and the autofluorescence image. Each pixel in the target image is determined based on the pixels in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image.

[0008] Secondly, embodiments of this application provide an optimization apparatus for multimodal image display, comprising:

[0009] The image acquisition module is used to acquire OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser.

[0010] An image fusion module is used to fuse the OCT image and the autofluorescence image to obtain a target image. Each pixel in the target image is determined based on each pixel in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image.

[0011] Thirdly, embodiments of this application provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the multimodal image display optimization method described in the first aspect above.

[0012] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal image display optimization method described in the first aspect.

[0013] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute the multimodal image display optimization method described in any of the first aspects above.

[0014] The beneficial effects of this application embodiment compared with the prior art are as follows: OCT images and autofluorescence images of the target vascular segment are acquired, and the acquired OCT images and autofluorescence images are fused to obtain the target image. Since OCT images have high resolution, determining each pixel in the target image based on each pixel in the OCT image ensures that the target image also has high resolution. Simultaneously, since near-infrared fluorescence imaging technology has deeper tissue penetration than OCT technology, determining the value of each pixel in the target image based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image allows the target image to more accurately reflect the tissue structure and the distribution of atherosclerotic plaques. Therefore, the target image not only possesses the high resolution and ease of identification of common atherosclerotic plaques characteristic of OCT images, but also overcomes the poor penetration problem of OCT technology, enabling the identification of the entire atherosclerotic plaque, thereby improving the accuracy of atherosclerotic plaque identification. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0016] Figure 1 This is a flowchart illustrating an optimization method for multimodal image display according to an embodiment of this application;

[0017] Figure 2 This is a schematic diagram of a target image containing atherosclerotic plaques provided in an embodiment of this application;

[0018] Figure 3 This is a schematic diagram of the direction of line A in the target image provided in the embodiments of this application;

[0019] Figure 4 This is a 3D carpet spread image with OCT image transparency of 0 provided in the embodiments of this application;

[0020] Figure 5 This is a 3D carpet spread image with 100% OCT image transparency provided in the embodiments of this application;

[0021] Figure 6 This is a schematic diagram of the structure of the multimodal image display optimization device provided in the embodiments of this application;

[0022] Figure 7 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0023] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0024] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0025] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0026] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0028] Example 1:

[0029] Figure 1 A flowchart illustrating an optimization method for multimodal image display according to an embodiment of the present invention is shown below, and is described in detail below:

[0030] Step S101: Obtain OCT images and autofluorescence images of the target vascular segment.

[0031] Among them, OCT technology utilizes an advanced fiber optic interferometer and a light source that can emit low-energy, broadband near-infrared light with a wavelength of 1320nm. Through miniaturized catheter technology, an imaging fiber is delivered into the artery, thereby providing a two-dimensional cross-sectional image of the artery, namely an OCT image.

[0032] Among them, the autofluorescence image is an image obtained by stimulating the autofluorescence of a specific patch component with a near-infrared laser. Near-infrared fluorescence imaging uses a laser source with a specific spectral range to irradiate fluorescent molecules. The fluorescent molecules are excited to emit photon signals with different spectral characteristics. The autofluorescence image is obtained by collecting and processing these photon signals.

[0033] Specifically, a segment of blood vessel to be scanned is designated as the target segment. An OCT instrument is used to scan this target segment, acquiring N consecutive OCT images, where N is an integer greater than or equal to 1. Simultaneously, a fluorescence laser imaging device is used to image the target segment, acquiring N consecutive autofluorescence images. For example, a segment of the coronary artery can be designated as the target segment, and both an OCT instrument and a fluorescence laser imaging device can be used to scan and image this target segment, acquiring 300 consecutive time-series OCT images and autofluorescence images of the target segment.

[0034] In this embodiment of the application, N frames of OCT images and autofluorescence images of the target blood vessel segment are acquired through corresponding acquisition methods. The obtained OCT images and autofluorescence images are time-series continuous images and the number is the same, which facilitates the subsequent fusion and display of the OCT images of the target blood vessel segment with the corresponding autofluorescence images.

[0035] Step S102: Fuse the above OCT image and the above autofluorescence image to obtain a target image. Each pixel in the target image is determined based on each pixel in the above OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the above autofluorescence image.

[0036] Specifically, based on the correspondence between each pixel in each OCT image and each pixel in the corresponding autofluorescence image, the pixel values ​​in the OCT image are replaced with the pixel values ​​in the corresponding autofluorescence image. Since the pixel values ​​in the autofluorescence image are autofluorescence intensity coefficients, replacing the pixel values ​​in the OCT image with the corresponding autofluorescence image values ​​is equivalent to replacing the pixel values ​​in the OCT image with the autofluorescence intensity coefficients in the autofluorescence image, thus obtaining the fused OCT-autofluorescence image, which is the target image.

[0037] It should be noted that when the OCT image and the autofluorescence image are the same size (i.e., have the same resolution), the pixel values ​​in the OCT image can be directly replaced with the corresponding pixel values ​​in the autofluorescence image based on the one-to-one correspondence between the pixels in the OCT image and the pixels in the autofluorescence image. However, when the OCT image and the autofluorescence image are not the same size, interpolation processing is performed on the OCT image and / or the autofluorescence image to obtain OCT images and autofluorescence images with the same resolution. Then, the OCT images and autofluorescence images with the same resolution are fused to avoid the difference in resolution between the OCT images and the autofluorescence images affecting the fusion and causing the target image to not meet the expected requirements.

[0038] In this embodiment, based on the correspondence between each pixel in the OCT image and each pixel in the corresponding autofluorescence image, the values ​​of the pixels in the OCT image are replaced with the values ​​of the pixels in the corresponding autofluorescence image to obtain the target image. Since the values ​​of the pixels in the autofluorescence image are autofluorescence intensity coefficients, which describe the distribution of atherosclerotic plaques, and the autofluorescence image has deep penetration, the target image obtained by fusing the OCT image and the autofluorescence image concentrates the feature representation of the OCT image and the autofluorescence image, thereby overcoming the problem of poor penetration of a single OCT image.

[0039] In this embodiment, an OCT image and an autofluorescence image of the target vascular segment are acquired. Based on the correspondence between the pixels of the OCT image and the autofluorescence image, the value of each pixel in the OCT image is replaced with the autofluorescence intensity coefficient of the corresponding pixel in the autofluorescence image, resulting in a target image that fuses the OCT image and the autofluorescence image. Since the pixel value of the autofluorescence image is the autofluorescence intensity coefficient, which describes the different conditions of atherosclerotic plaques, and near-infrared fluorescence imaging technology has deep tissue penetration, the target image that fuses the OCT image and the autofluorescence image not only retains the high resolution and clear plaque imaging characteristics of the OCT image, but also incorporates the strong tissue penetration characteristics of the autofluorescence image, thus concentrating the feature representation of the OCT image and the autofluorescence image. This allows the target image to identify the whole picture of the atherosclerotic plaque, thereby improving the accuracy of identifying atherosclerotic plaques.

[0040] In some embodiments, after step S101 described above, the method further includes:

[0041] The obtained autofluorescence image was then stained.

[0042] Correspondingly, step S102 specifically includes:

[0043] The target image is obtained by fusing the above OCT image with the autofluorescence image after staining.

[0044] Specifically, because fluorescent molecules in different biological tissues (such as atherosclerotic plaques) are excited to emit photon signals with different spectral characteristics, and different photon signals can produce different display features, when biological tissues are different, such as in the autofluorescence image of a vascular segment containing atherosclerotic plaques, the characteristics of the atherosclerotic plaques displayed are also different from those of normal biological tissues.

[0045] In this embodiment, staining processing can be performed based on the characteristics of plaques displayed in the autofluorescence image to enhance the display of atherosclerotic plaque features. For example, a gradient of different colors can be used to describe the autofluorescence intensity coefficient of the autofluorescence image; a larger fluorescence coefficient indicates a greater content and severity of atherosclerotic plaques. For instance, if a gradient from blue to red to yellow is used to describe the autofluorescence intensity coefficient of the autofluorescence image, the closer the gradient is to blue, the smaller the autofluorescence intensity coefficient, and the smaller the content and severity of atherosclerotic plaques; conversely, the closer it is to yellow, the larger the autofluorescence intensity coefficient, and the greater the content and severity of atherosclerotic plaques.

[0046] In this embodiment, the distribution of atherosclerotic plaques is described by using different color gradients, making the distribution of atherosclerotic plaques more intuitive and clear, thus enabling users (such as doctors) to easily identify atherosclerotic plaques.

[0047] In some embodiments, after step S101, the method further includes:

[0048] The OCT images and the autofluorescence images were subjected to noise reduction processing.

[0049] Correspondingly, step S102 specifically includes:

[0050] The target image is obtained by fusing the denoised OCT image and the denoised autofluorescence image.

[0051] Specifically, due to problems with the imaging system, random interference from the external environment, and issues with infrared detectors, especially since OCT technology is a high-resolution technology with high requirements for changes in the external environment and the signal data acquisition process, the acquired OCT images and autofluorescence images inevitably contain some noise, thus affecting image quality. To reduce the impact of noise on image quality and obtain accurate and clear OCT and autofluorescence images, this application embodiment, after obtaining the OCT and autofluorescence images, can perform noise reduction processing on the OCT and autofluorescence images using methods such as time-domain denoising, frequency-domain denoising, or a combination of time-domain and frequency-domain denoising. For example, wavelet transform denoising processing is performed on the OCT and / or autofluorescence images based on the following steps: performing two-dimensional discrete wavelet decomposition on the OCT and / or autofluorescence images to obtain wavelet coefficients at different resolutions; performing threshold quantization processing on the high-frequency coefficients in the obtained wavelet coefficients; and calculating wavelet reconstruction of the two-dimensional signal based on the low-frequency coefficients obtained from wavelet decomposition and the thresholded high-frequency coefficients.

[0052] In this embodiment, after obtaining the OCT image and autofluorescence image of the target blood vessel segment, the OCT image and autofluorescence image are subjected to corresponding noise reduction processing to reduce the impact of noise on image quality, thereby improving the accuracy of image recognition and diagnosis.

[0053] In some embodiments, after step S102 described above, the method further includes:

[0054] A1. Vectorize the N frames of target images of the target blood vessel segment to obtain a carpet spread image, where N is an integer greater than or equal to 1. The carpet spread image describes the overall features of the target blood vessel segment.

[0055] Specifically, since the number of frames of target images of the target blood vessel segment is generally large, users need to identify and diagnose atherosclerotic plaques from a large number of target images, resulting in a significant workload. Therefore, after obtaining N frames of target images of the target blood vessel segment, these N frames are vectorized and merged into a single frame to obtain a carpet spread image. This carpet spread image is then displayed by fusing the N frames of target images of the target blood vessel segment.

[0056] In this embodiment, since the blanket spread image incorporates all target images of the target vascular segment, it reflects the overall characteristics of the target vascular segment, enabling users to easily and intuitively identify and diagnose it through the blanket spread image, thereby reducing workload and diagnosis time.

[0057] In some embodiments, prior to step A1 above, the method further includes:

[0058] B1. Perform interpolation processing on the above target image to obtain the interpolated pixels of the above target image.

[0059] B2. Project the target image based on the above pixels to obtain a new target image.

[0060] Correspondingly, step A1 above specifically includes:

[0061] The N new target images of the target blood vessel segment are vectorized to obtain the carpet spread image.

[0062] Specifically, since the resolution of the target image may differ from the resolution required for subsequent image processing, interpolation is necessary to obtain the interpolated pixels. Then, the original target image (the uninterpolated image) is projected to obtain a new target image with the required resolution. For example, if the target image size is 300×500, bilinear interpolation is used. This involves linearly interpolating the original target image using its four existing adjacent pixels in both the x and y directions to obtain the interpolated pixels. The original target image is then projected based on these interpolated pixels to obtain a target image with a resolution of 300×500 in polar coordinates.

[0063] In this embodiment of the application, since the target image is interpolated, the target image with the required resolution size can be obtained for subsequent image processing, which facilitates the subsequent processing of the target image.

[0064] In some embodiments, step A1 above includes:

[0065] C1. Obtain the maximum fluorescence coefficient of each target image on line A, and generate the maximum fluorescence coefficient vector corresponding to each target image. Line A refers to the signal scan line when the autofluorescence instrument scans and images, and the fluorescence coefficient is the autofluorescence intensity coefficient corresponding to each pixel in the target image.

[0066] C2. Based on the maximum vector of each fluorescence coefficient, merge them to generate the target matrix.

[0067] C3. Obtain the carpet spread diagram based on the above target matrix.

[0068] Specifically, the A-line refers to the signal scan line during autofluorescence imaging. The direction of the signal scan line determines the temporal position of the acquired autofluorescence image. Since the target image is a cross-sectional image of the target blood vessel segment, the blanket spread image unfolds the inner wall of the target blood vessel segment to observe its overall features. Therefore, it is necessary to merge the unfolded inner wall representations of each frame of the target image into a blanket spread image. By calculating the maximum fluorescence coefficient on each A-line of the target image, which describes the autofluorescence intensity coefficient of the target image on each A-line, the maximum fluorescence coefficient vector of each target image is obtained based on the maximum fluorescence coefficient. This maximum fluorescence coefficient vector describes the characteristic representation of the unfolded inner wall of the target image. Thus, the maximum fluorescence coefficient vectors of each frame of the target image are merged according to the temporal position to obtain the target matrix, i.e., the blanket spread image. If the maximum fluorescence coefficient vector is calculated based on other directions to obtain the blanket spread image, the resulting blanket spread image cannot correspond to the inner wall of the target blood vessel segment, and the feature information of the resulting blanket spread image may be disordered. Figure 2 The above, Figure 2 For a single frame of the target image, 201 represents the characteristic features of atherosclerotic plaques, such as... Figure 3As shown, 301 represents the direction of line A in the target image, which is perpendicular to the vessel wall. That is, the target vessel segment is cylindrical, and line A is equivalent to the radius in the cross-section of the vessel segment. The maximum fluorescence coefficient vector of the target image is obtained by calculating the maximum fluorescence coefficient of each line A. This is equivalent to compressing the pixels on line A of the target image into a circle according to the direction of line A, and then splitting this circle into a line (i.e., the maximum fluorescence coefficient vector). This line is the inner wall unfolding representation of the target image in this frame. Since the maximum fluorescence coefficient is calculated according to the direction of line A, the feature representation of atherosclerotic plaque 201 is completely mapped at this time. Due to the feature representation of the target image, when compressing in other directions, the feature representation of atherosclerotic plaque 201 may be misaligned, and the maximum fluorescence coefficient vector cannot correctly describe the features of atherosclerotic plaque 201. This may result in information errors in the carpet spread image calculated in other directions, and it may not correspond to the inner wall of the target vessel segment. Therefore, it is necessary to calculate and obtain the carpet spread image by calculating the maximum fluorescence coefficient of line A. For example, assuming the target image consists of 200 frames, and each target image is 300×500 pixels, then there are 500 A-lines in the target image, each with 300 pixels. Calculating the maximum fluorescence coefficient vector on each A-line of a frame yields a 1×500 maximum fluorescence coefficient vector. After processing the 200 frames, 200 1×500 maximum fluorescence coefficient vectors are obtained. These maximum fluorescence coefficient vectors are then merged according to their temporal positions within the target image to obtain a 200×500 target matrix. Based on this target matrix, a 200×500 carpet spread image is obtained.

[0069] In this embodiment, by obtaining the maximum fluorescence coefficient of each A-line of each target image, the maximum fluorescence coefficient vector of each target image is obtained. According to the corresponding temporal position, the maximum fluorescence coefficient vectors are merged into a target matrix to obtain a carpet spread image. Since the maximum fluorescence coefficient is calculated based on the autofluorescence intensity coefficient of all pixels on each A-line of the target image, the maximum fluorescence coefficient vector of the target image obtained based on the maximum fluorescence coefficient describes the features of each A-line in the target image. Therefore, the carpet spread image generated by merging the maximum fluorescence coefficient vectors of all target images of the target blood vessel segment displays the overall features of the target blood vessel segment, enabling users to identify atherosclerotic plaques in the target blood vessel segment through a single-frame carpet spread image, thereby reducing the user's workload.

[0070] In some embodiments, step C2 above includes:

[0071] The two consecutive maximum fluorescence coefficient vectors are filtered.

[0072] The maximum vectors of the various fluorescence coefficients after filtering are merged to generate the target matrix.

[0073] Specifically, when acquiring autofluorescence images, temporally continuous autofluorescence images are obtained. However, blood vessels themselves are continuous tissues in the longitudinal direction. Therefore, there may be slight gaps between two consecutive autofluorescence images, causing discontinuity between the two consecutive target images obtained through fusion. During autofluorescence imaging, the fluorescence probe of the autofluorescence instrument is pulled back to image, and the imaging is performed in the cross-sectional direction of the blood vessel. Photon signals from different biological tissues, including atherosclerosis, are acquired in the cross-sectional direction of the blood vessel. The photon signals in the longitudinal direction are temporally connected without other correlations. Therefore, the obtained consecutive N frames of autofluorescence images lose their longitudinal continuity. Subsequently, when merging to generate a carpet spread image, the resulting carpet spread image is merely a stack of cross-sectional images, without continuous correlation with adjacent images, thus making the generated carpet spread image incomplete. Therefore, in order to establish the vertical connectivity of the autofluorescence images, when synthesizing the carpet spread image, every two consecutive maximum fluorescence coefficient vectors are filtered, such as by bilateral filtering, and the photon signal of the current frame image is corrected by the autofluorescence image of the neighborhood. Then, the maximum fluorescence coefficient vectors after each filtering process are merged to generate the target matrix.

[0074] In this embodiment, since the autofluorescence instrument is a cross-sectional imaging instrument, there may be slight gaps between two consecutive frames of the autofluorescence image of the target blood vessel segment. This results in slight gaps in biological tissues such as atherosclerotic plaques in the directly generated carpet image. Therefore, the maximum vector of every two consecutive fluorescence coefficients is filtered to fit the gaps, thereby ensuring continuity between the two consecutive frames and thus giving the obtained carpet image continuous information and more accurate feature representation.

[0075] In some embodiments, after step A1 above, the method further includes:

[0076] Based on the above carpet spread image, the above target blood vessel segment is reconstructed in three dimensions to obtain the 3D carpet spread image of the above target blood vessel segment.

[0077] Specifically, since blood vessels have a near-cylindrical structure, when acquiring cross-sectional images of blood vessels, the optical probe rotates its rotating head while simultaneously translating longitudinally to obtain pull-back data, i.e., autofluorescence images. However, these autofluorescence images are typically two-dimensional cross-sectional images. Because some useful information may be lost during the processing and recognition of two-dimensional images due to angle and other issues, the resulting carpet image is rolled downwards based on the target blood vessel segment structure. The positional offset of pixels at different distances from the axis is calculated based on the curvature after rolling, obtaining the corresponding point and depth information of each pixel in the carpet image. Interpolation smoothing is then performed based on this positional offset to establish a three-dimensional model of the target blood vessel segment. Finally, the carpet image undergoes texture mapping and other processing to obtain a 3D carpet image, allowing users to adjust the viewing angle to obtain the required information.

[0078] In this embodiment, a 3D atherosclerotic plaque image is obtained by reconstructing the 2D atherosclerotic plaque image and the cylindrical structure of the target vessel end itself. Since some useful information may be lost when identifying and diagnosing a 2D atherosclerotic plaque image, the 3D atherosclerotic plaque image allows the user to adjust the viewing angle according to their needs and closely resembles the actual target vessel segment. Therefore, the 3D atherosclerotic plaque image obtained through 3D reconstruction enables users to more intuitively observe the distribution of atherosclerotic plaques throughout the target vessel segment, thereby improving the accuracy of identifying and diagnosing atherosclerotic plaques.

[0079] In some embodiments, the above-described method for optimizing multimodal image display further includes:

[0080] The 3D carpet image above is gradient-set using a gradient opacity function, which is used to determine the opacity corresponding to different gradient values.

[0081] Specifically, since the image features reflected by image data often differ depending on the transparency, and different image features can affect the user's judgment of the image data, that is, the user's judgment is related to the transparency of the displayed image data. Therefore, by using a gradient opacity function to set the gradient of the above 3D carpet image, different degrees of transparency visualization of the organization can be achieved, and an adjustable gradient interactive interface can be provided, allowing users to adjust the transparency of the OCT image according to their needs, highlighting the organization that the user is concerned about, which is conducive to improving the accuracy of the user's judgment of the image data.

[0082] For example, such as Figure 4 The image shown is a 3D carpet spread with the OCT image transparency set to 0. In this case, the OCT image transparency is 0, meaning no transparency processing is applied, and the OCT image features are clearly visible. Users can adjust the OCT image transparency using the provided gradient adjustment interface, as needed. Figure 5The OCT image shown is a 3D carpet spread with 100% transparency. At this time, the 3D carpet spread only displays the image features of the autofluorescent image, which is equivalent to a 3D carpet spread of the autofluorescent image.

[0083] In this embodiment, a gradient opacity function is used to display OCT image features with different transparency in a 3D carpet spread image. This allows the transparency of OCT image features to be adjusted according to the user's needs, thereby facilitating the analysis of the distribution and severity of atherosclerotic plaques.

[0084] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0085] Example 2:

[0086] Corresponding to the optimization method for multimodal image display described in the above embodiments, Figure 6 A structural block diagram of a multimodal image pixel optimization device provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0087] Reference Figure 6 The device includes: an image acquisition module 61 and an image fusion module 62. Among them,

[0088] The image acquisition module 61 is used to acquire OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser.

[0089] Image fusion module 62 is used to fuse the OCT image and the autofluorescence image to obtain a target image. Each pixel in the target image is determined based on each pixel in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image.

[0090] In this embodiment, an OCT image and an autofluorescence image of the target vascular segment are acquired. Based on the correspondence between the pixels of the OCT image and the autofluorescence image, the value of each pixel in the OCT image is replaced with the autofluorescence intensity coefficient of the corresponding pixel in the autofluorescence image, resulting in a target image that fuses the OCT image and the autofluorescence image. Since the pixel value of the autofluorescence image is the autofluorescence intensity coefficient, which describes the different conditions of atherosclerotic plaques, and near-infrared fluorescence imaging technology has deep tissue penetration, the target image that fuses the OCT image and the autofluorescence image not only retains the high resolution and clear plaque imaging characteristics of the OCT image, but also incorporates the strong tissue penetration characteristics of the autofluorescence image, thus concentrating the feature representation of the OCT image and the autofluorescence image. This allows the target image to identify the whole picture of the atherosclerotic plaque, thereby improving the accuracy of identifying atherosclerotic plaques.

[0091] In some embodiments, the image acquisition module 61 further includes:

[0092] The autofluorescence image staining unit is used to stain the obtained autofluorescence image.

[0093] In some embodiments, the above-described multimodal image display optimization apparatus further includes:

[0094] The noise reduction module is used to perform noise reduction processing on the above-mentioned OCT image and the above-mentioned autofluorescence image.

[0095] In some embodiments, the above-described multimodal image display optimization apparatus further includes:

[0096] The image processing module is used to vectorize the N frames of target images of the target blood vessel segment to obtain a carpet spread image, where N is an integer greater than or equal to 1. The carpet spread image describes the overall features of the target blood vessel segment.

[0097] In some embodiments, the image processing module further includes:

[0098] The interpolation unit is used to perform interpolation processing on the target image to obtain the interpolated pixels of the target image.

[0099] The projection unit is used to project the target image based on the aforementioned pixels to obtain a new target image.

[0100] In some embodiments, the image processing module further includes:

[0101] The maximum vector calculation unit is used to obtain the maximum fluorescence coefficient of each target image on line A and generate the maximum fluorescence coefficient vector corresponding to each target image. The fluorescence coefficient is the autofluorescence intensity coefficient corresponding to each pixel in the target image.

[0102] The target matrix acquisition unit is used to merge and generate a target matrix based on the maximum fluorescence coefficient vector of each of the above target images.

[0103] The carpet spread image acquisition unit is used to obtain the carpet spread image based on the above target matrix.

[0104] In some embodiments, the target matrix acquisition unit includes:

[0105] The filtering unit is used to filter two consecutive maximum fluorescence coefficient vectors.

[0106] The merging unit is used to merge the maximum vectors of the various fluorescence coefficients after filtering to generate the target matrix.

[0107] In some embodiments, the above-described multimodal image display optimization apparatus further includes:

[0108] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction of the target vascular segment based on the above carpet spread image, and obtain the 3D carpet spread image of the target vascular segment.

[0109] In some embodiments, the above-described multimodal image display optimization apparatus further includes:

[0110] The transparency display module is used to apply a gradient opacity function to the 3D carpet image to determine the opacity corresponding to different gradient values.

[0111] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0112] Example 3:

[0113] Figure 7 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Figure 7 As shown, the terminal device 7 of this embodiment includes: at least one processor 70 ( Figure 7 The diagram shows only one processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70. When the processor 70 executes the computer program 72, it implements the steps of any of the above-described method embodiments, for example... Figure 1Steps S11 to S12 are shown. Alternatively, when the processor 70 executes the computer program 72, it implements the functions of each module / unit in the above-described devices, such as... Figure 6 The functions of modules 61 to 62 are shown.

[0114] For example, the computer program 72 can be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 72 in the electronic device 7. For example, the computer program 72 can be divided into an image acquisition module 61 and an image fusion module 62, with the specific functions of each module as follows:

[0115] The image acquisition module 61 is used to acquire OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser.

[0116] Image fusion module 62 is used to fuse the OCT image and the autofluorescence image to obtain a target image. Each pixel in the target image is determined based on each pixel in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image.

[0117] The terminal device 7 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This terminal device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 The example of terminal device 7 is merely an illustration and does not constitute a limitation on terminal device 7. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0118] The processor 70 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0119] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. In other embodiments, the memory 71 may be an external storage device of the terminal device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the terminal device 7. Furthermore, the memory 71 may include both internal and external storage units of the terminal device 7. The memory 71 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0120] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0121] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0122] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0123] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.

[0124] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0126] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0127] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0128] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0129] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An optimization method for multimodal image display, characterized in that, include: Obtain OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser. The target image is obtained by fusing the OCT image and the autofluorescence image. Each pixel in the target image is determined based on the pixels in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image. The N frames of target images of the target blood vessel segment are vectorized to obtain a carpet spread image, where N is an integer greater than or equal to 1. The carpet spread image describes the overall features of the target blood vessel segment. The step of fusing the OCT image and the autofluorescence image to obtain the target image includes: Based on the correspondence between each pixel in the OCT image and each pixel in the autofluorescence image, the values ​​of the pixels in the OCT image are replaced with the values ​​of the pixels in the autofluorescence image to obtain the target image; The step of vectorizing N frames of target images of the target blood vessel segment to obtain a carpet spread image includes: The maximum fluorescence coefficient of each target image on line A is obtained, and the maximum fluorescence coefficient vector corresponding to each target image is generated. Line A refers to the signal scan line when the autofluorescence instrument scans and images, and the fluorescence coefficient is the autofluorescence intensity coefficient corresponding to each pixel in the target image. Based on the maximum vector of each fluorescence coefficient, the target matrix is ​​generated by merging them. The carpet spread diagram is obtained based on the target matrix.

2. The optimization method for multimodal image display as described in claim 1, characterized in that, Before vectorizing the N frames of target images of the target blood vessel segment, the method further includes: The target image is interpolated to obtain the interpolated pixels of the target image; The target image is projected based on the pixels to obtain a new target image; The step of vectorizing the N frames of target images of the target blood vessel segment includes: The N frames of new target images of the target blood vessel segment are vectorized.

3. The optimization method for multimodal image display as described in claim 1, characterized in that, The step of merging the target matrix based on the maximum vectors of the fluorescence coefficients includes: Filter the two consecutive maximum fluorescence coefficient vectors; The maximum vectors of the various fluorescence coefficients after filtering are merged to generate the target matrix.

4. The optimization method for multimodal image display as described in claim 1, characterized in that, After obtaining the carpet spread diagram, the following is also included: Based on the carpet spread image, a three-dimensional reconstruction of the target blood vessel segment is performed to obtain a 3D carpet spread image of the target blood vessel segment.

5. The optimization method for multimodal image display as described in claim 4, characterized in that, Also includes: The 3D carpet image is gradient-set using a gradient opacity function, which is used to determine the opacity corresponding to different gradient values.

6. An optimization device for multimodal image display, characterized in that, include: The image acquisition module is used to acquire OCT images and autofluorescence images of the target vascular segment. The OCT images are obtained by optical coherence tomography, and the autofluorescence images are obtained by stimulating the autofluorescence of specific plaque components with near-infrared laser. An image fusion module is used to fuse the OCT image and the autofluorescence image to obtain a target image. Each pixel in the target image is determined based on each pixel in the OCT image, and the value of each pixel in the target image is determined based on the autofluorescence intensity coefficient of the pixels in the autofluorescence image. The image processing module is used to vectorize N frames of target images of the target blood vessel segment to obtain a carpet spread image, where N is an integer greater than or equal to 1, and the carpet spread image describes the overall features of the target blood vessel segment. The image fusion module is specifically used to: replace the values ​​of the pixels in the OCT image with the values ​​of the pixels in the autofluorescence image according to the correspondence between each pixel in the OCT image and each pixel in the autofluorescence image, thereby obtaining the target image; The image processing module includes: The maximum vector calculation unit is used to obtain the maximum fluorescence coefficient of each target image on the A line and generate the maximum fluorescence coefficient vector corresponding to each target image. The A line refers to the signal scan line when the autofluorescence instrument scans and images, and the fluorescence coefficient is the autofluorescence intensity coefficient corresponding to each pixel in the target image. The target matrix acquisition unit is used to merge and generate a target matrix based on the maximum vectors of the fluorescence coefficients. The carpet spread image acquisition unit is used to obtain the carpet spread image based on the target matrix.

7. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images