A multimodal intracavitary imaging system and a multimodal data fusion method

Abstract

The present application provides a multimodal intracavitary imaging system and a multimodal data fusion method. The multimodal intracavitary imaging system includes: a first swept-frequency light source, a second swept-frequency light source, a first fiber coupler, a time division multiplexer and an imaging catheter; the first swept-frequency light source is connected to the first fiber coupler; the sample arm of the first fiber coupler is connected to the imaging catheter optical signal through the time division multiplexer; the second swept-frequency light source is connected to the imaging catheter optical signal through the time division multiplexer; wherein, the first swept-frequency light source and the second swept-frequency light source can emit light of different central wavelengths; the imaging catheter can detect and image the intracavitary tissue based on the light emitted by the first swept-frequency light source and the second swept-frequency light source. Based on the multimodal intracavitary imaging system and multimodal data fusion method provided by the present application, the lipid level of lipid plaques in the patient's blood vessels can be accurately assessed.

Description

Technical Field

[0001] The present application relates to the field of medical devices and signal processing technology, and in particular to a multimodal intracavitary imaging system and a multimodal data fusion method. Background Art

[0002] Most cases of acute coronary syndromes are caused by the rupture of thin cystic fibrous aneurysms (vulnerable plaques), leading to the formation of blood clots within the coronary arteries. Timely and accurate identification of vulnerable plaques within a patient's coronary arteries and analysis and evaluation of their distribution can help physicians develop effective treatment plans for their patients.

[0003] Optical coherence tomography (OCT) is based on the principle of bottom-coherent interference to obtain tomographic capabilities in the depth direction. Through scanning, it can reconstruct two-dimensional or three-dimensional images of the internal structure of biological tissues or materials. Its signal contrast comes from the spatial variation of the optical reflection or scattering properties within the biological tissues or materials. OCT can identify the structural features of coronary plaques that cannot be accurately identified by angiography or intravascular ultrasound. Near-infrared spectroscopy (NIRS) is a new intravascular imaging technology that can provide chemical assessments related to the presence of cholesterol esters in the lipid core and can distinguish cholesterol from collagen in tubular plaques through its spectral fingerprint. NIRS can accurately identify lipid-rich plaques in human tissues.

[0004] However, the above-mentioned evaluation method based on single-modality images can only evaluate vulnerable plaques from the perspective of structure or composition, cannot accurately determine the lipid content of vulnerable plaques, and is not conducive to physicians proposing adaptive treatment plans. Summary of the Invention

[0005] The embodiments of the present application provide a lipid plaque assessment system and method based on multimodal intraluminal imaging, which can accurately assess the lipid content of lipid plaques in patients' blood vessels.

[0006] In a first aspect, an embodiment of the present application provides a multimodal intracavity imaging system, the system comprising: a first swept-frequency light source, a second swept-frequency light source, a first fiber coupler, a time division multiplexer, and an imaging catheter;

[0007] The first swept-frequency light source is connected to the first fiber coupler; the sample arm of the first fiber coupler is connected to the imaging catheter optical signal through the time division multiplexer; the second swept-frequency light source is connected to the imaging catheter optical signal through the time division multiplexer;

[0008] The first swept-frequency light source and the second swept-frequency light source can emit light of different central wavelengths; the imaging catheter can detect and image the intracavitary tissue based on the light emitted by the first swept-frequency light source and the second swept-frequency light source.

[0009] The multimodal intracavitary imaging system provided herein employs a first swept-frequency light source and a second swept-frequency light source capable of emitting light of different central wavelengths. After light from the first swept-frequency light source is split by a first fiber coupler, the sample light passes through a time-division multiplexer into an imaging catheter for detecting intravascular tissue. Meanwhile, light from the second swept-frequency light source passes through a time-division multiplexer into an imaging catheter for detecting intravascular tissue. Thus, the first swept-frequency light source can identify intravascular surface microstructures, while the second swept-frequency light source can assess lipid content in corresponding regions of the vascular wall. Compared to existing technologies, lipid content assessment can utilize light with a central wavelength having a superior absorption peak, thereby improving the reproducibility of imaging for lipid plaque or vulnerable plaque identification.

[0010] Optionally, the imaging catheter includes two imaging probes, one of the two imaging probes is connected to the first swept-frequency light source and the second swept-frequency light source respectively;

[0011] The imaging probes respectively connected to the first swept-frequency light source and the second swept-frequency light source can be used to irradiate the light emitted by the first swept-frequency light source and the second swept-frequency light source onto the intracavitary tissue and receive echo signals; the other of the two imaging probes is used to receive the echo signal of the second swept-frequency light source.

[0012] Based on this optional approach, one imaging probe irradiates the vascular tissue with light from the first and second swept-frequency light sources, while the other imaging probe receives the echo light from the lipid plaque. This prevents interference between the echo light and the incident light, thereby improving the accuracy of the detection imaging and the reproducibility of the assessment results.

[0013] Optionally, the multimodal intracavitary imaging system further comprises a rotary joint connected between the time division multiplexer and the imaging catheter, and the echo signals of the two imaging probes can be transmitted in split optical paths through the rotary joint.

[0014] Optionally, the multimodal intracavity imaging system further comprises a first fiber optic circulator connected between the time division multiplexer and the rotary joint, wherein one port of the first fiber optic circulator is connected to the time division multiplexer, a second port of the first fiber optic circulator is connected to a first port of the rotary joint, and a second port of the rotary joint is connected to the imaging catheter;

[0015] The echo of the imaging probe connected to the first swept-frequency light source can enter the first optical fiber circulator through the second port of the first optical fiber circulator and be transmitted outward from the third port of the first optical fiber circulator.

[0016] Based on the above optional method, the incident light can only propagate in one direction in the first fiber optic circulator, that is, the light entering from port one can only be emitted from port two, and the light entering from port two can only be emitted from port three, which can effectively avoid interference between the light emitted by the first swept-frequency light source and the echo light obtained by the imaging probe.

[0017] Optionally, the multimodal intracavity imaging system further comprises a second fiber circulator, a collimator, a mirror, a second fiber coupler and a balanced photodetector;

[0018] The reference arm of the first fiber optic coupler is connected to the collimator through the second fiber optic circulator. The light emitted by the collimator can be reflected by the mirror and transmitted outward from the third port of the second fiber optic circulator through the second port of the second fiber optic circulator. The third port of the first fiber optic circulator and the third port of the second fiber optic circulator are connected to the balanced photodetector through the second fiber optic coupler. The balanced photodetector is used to convert the optical signal output by the second fiber optic coupler into an electrical signal.

[0019] Based on this optional approach, after the light emitted by the first swept-frequency light source is split by the first fiber coupler, the reference light then passes through the second fiber circulator and collimator, and is reflected by a mirror. By leveraging the fact that the incident light can only propagate in one direction within the second fiber circulator, interference between the incident reference light and the reflected light from the mirror is avoided. By providing a balanced photodetector, the effects of receiver noise and electronic circuit noise on weak light signal detection can be reduced, thereby improving imaging accuracy.

[0020] Optionally, the multimodal intracavity imaging system further includes a control device connected to the balanced photodetector and, via the photodetector, to the third port of the rotary joint. The control device is configured to analyze and display the electrical signal output by the balanced photodetector, and to analyze and display the echo of the imaging probe. The control device is respectively connected to the first swept-frequency light source and the second swept-frequency light source, and controls the alternating activation of the first and second swept-frequency light sources.

[0021] Based on the above optional approach, the control device controls the first and second swept-frequency light sources to emit light at different time periods, thereby preventing interference between the light emitted by the first and second swept-frequency light sources. The control device can also process the electrical signals generated by the photodetector and the balanced photodetector, converting them into images or other data and displaying them as images to guide physicians in analyzing vulnerable plaques within blood vessels.

[0022] In a second aspect, an embodiment of the present application provides a multimodal data fusion method, the method comprising: acquiring multiple OCT image sequences and NIRS image sequences of intracavitary tissue acquired by the multimodal intracavitary imaging system described in the first aspect above, wherein the OCT image sequence comprises multiple OCT images, and the NIRS image sequence comprises multiple NIRS images corresponding one-to-one to the multiple OCT images;

[0023] Each of the OCT images and the NIRS image corresponding to the OCT image are registered; the lipid plaque area in each of the OCT images is identified, and a first judgment value of each of the OCT images is determined based on the area of ​​the lipid plaque area of ​​the multiple OCT images; the second judgment value of each of the NIRS images is determined based on the sampling frequency used when acquiring the NIRS image sequence and the number of samples of each of the NIRS images, where the number of samples of the NIRS images is the number of absorption values ​​greater than a threshold value; the first judgment value of each of the OCT images and the second judgment value of the NIRS image corresponding to each of the OCT images are fused to determine the lipid level indicated by each of the OCT images.

[0024] Based on the multimodal data fusion method provided by the present application, each OCT image is first registered one by one with its corresponding NIRS image, and then the size and structure of the plaque in the blood vessel are determined by identifying the lipid plaque area contained in the OCT image, and the first judgment value of each OCT image is determined based on the area of ​​the lipid plaque area of ​​the multiple OCT images. At the same time, based on the absorption value of the spectrum by the tissue in the blood vessel displayed in the NIRS image, it is determined whether the tissue in the blood vessel is lipid, thereby determining the second judgment value of each NIRS image based on the sampling frequency and the number of samples of each NIRS image. By fusing the first judgment value and the second judgment value, the lipid level indicated by each of the OCT images can be determined. Compared with the existing method of evaluating lipid plaques or vulnerable plaques based on single-modality images, the present application utilizes the complementarity of OCT images and NIRS images to fuse image data of multiple modalities, thereby evaluating the lipid level of lipid plaques and improving the accuracy of the evaluation.

[0025] Optionally, registering each of the OCT images with the NIRS image corresponding to the OCT image includes:

[0026] For each of the OCT images and the NIRS images corresponding to the OCT images, the OCT images and the NIRS images are respectively input into a trained target detection network to identify the position of the guide wire in the OCT images and the NIRS images. According to the offset angle of the guide wire position, the rotation angle of the pixel matrix of the OCT image relative to the pixel matrix of the NIRS image is adjusted to align the OCT image with the NIRS image.

[0027] Based on the above optional method, the degree of offset of the OCT image relative to the NIRS image is determined according to the position of the guidewire in each OCT image and its corresponding NIRS image, and the rotation angle of the OCT image relative to the NIRS image is adjusted according to the rotation angle of the guidewire in the two images, so that each OCT image and its corresponding NIRS image are registered one by one, thereby facilitating subsequent data fusion.

[0028] Optionally, identifying the lipid plaque area in each of the OCT images includes:

[0029] Inputting each of the OCT images into a trained U-Net network to obtain a first lipid segmentation mask for each of the OCT images;

[0030] Obtaining a multidimensional feature vector of each of the OCT images, and inputting the multidimensional feature vector into a trained random forest to obtain a second lipid segmentation mask for each of the OCT images;

[0031] The lipid plaque area in each of the OCT images is determined according to the first lipid segmentation mask and the second lipid segmentation mask of each of the OCT images.

[0032] Based on the above optional method, the semantic segmentation algorithm and the traditional digital image processing method are combined. Multiple lipid segmentation masks of the OCT image are obtained respectively through multiple algorithms, and then the multiple lipid segmentation masks are operated with AND operation to identify the lipid plaque area in the OCT image, which can improve the accuracy of lipid plaque area identification.

[0033] Optionally, determining the first judgment value of each of the OCT images according to the area of ​​the lipid plaque region of the plurality of OCT images includes:

[0034] Obtain the area of ​​the lipid plaque region in the multiple OCT images; divide the area of ​​the lipid plaque region in each of the OCT images by the maximum value of the areas of the lipid plaque regions in the multiple OCT images to obtain a first judgment value for each of the OCT images.

[0035] Optionally, determining the second judgment value of each NIRS image according to a sampling frequency used when acquiring the NIRS image sequence and the number of samples of each NIRS image includes:

[0036] For each of the NIRS images, the number of samples of the NIRS image is divided by a sampling frequency used when acquiring the plurality of NIRS image sequences to obtain a second determination value for each of the NIRS images.

[0037] Optionally, fusing the first determination value of each of the OCT images with the second determination value of the NIRS image corresponding to each of the OCT images to determine the lipid level indicated by each of the OCT images includes:

[0038] For each of the OCT images, determining a support matrix according to a first determination value of the OCT image and a second determination value of the NIRS image corresponding to the OCT image;

[0039] Obtaining a weight coefficient according to the eigenvector corresponding to the maximum eigenvalue of the support matrix;

[0040] performing weighted summation on the first judgment value of the OCT image and the second judgment value of the NIRS image corresponding to the OCT image according to the weight coefficient to obtain a fusion value;

[0041] The fusion value is input into a Kalman filter to obtain an evaluation parameter of each of the OCT images, and the evaluation parameter is used to characterize the lipid level.

[0042] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above-mentioned second aspects is implemented.

[0043] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any of the methods described in the second aspect above.

[0044] In the fifth aspect, an embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and runnable on the at least one processor, and when the processor executes the computer program, it implements any one of the methods described in the second aspect above.

[0045] It can be understood that the beneficial effects of the third to fifth aspects mentioned above can be found in the relevant descriptions of the first and second aspects mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a block diagram of the overall structure of the intravascular imaging system provided by one embodiment of the present application;

[0048] Figure 2 This is a block diagram of the internal structure of a rotary joint provided in one embodiment of the present application;

[0049] Figure 3 This is an embodiment of the present application based on Figure 1 The system shown here acquires an image of tissue within the blood vessel cavity;

[0050] Figure 4 This is a flowchart of a multimodal data fusion method provided in one embodiment of the present application;

[0051] Figure 5 This is a multimodal data fusion result diagram provided in an embodiment of the present application.

[0052] Explanation of the accompanying drawings: 101, first swept-frequency light source; 102, second swept-frequency light source; 103, first fiber optic coupler; 104, time-division multiplexer; 105, imaging catheter; 1051, imaging probe; 106, rotary joint; 1061, third fiber optic coupler; 107, first fiber optic circulator; 108, photodetector; 109, control device; 1091, signal processing and display device; 1092, control device; 110, second fiber optic circulator; 111, collimator; 112, mirror; 113, second fiber optic coupler; 114, balanced photodetector; 200, blood vessel wall; 300, lipid plaque. DETAILED DESCRIPTION

[0053] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0054] Most cases of acute coronary syndromes are caused by the rupture of thin cystic fibrous aneurysms (vulnerable plaques), leading to the formation of blood clots within the coronary arteries. Timely and accurate identification of vulnerable plaques within a patient's coronary arteries and analysis and evaluation of their distribution can help physicians develop effective treatment plans for their patients.

[0055] OCT is based on the principle of low-coherence interference to obtain tomographic images in the depth direction. Through scanning, it can reconstruct a two-dimensional or three-dimensional image of the internal structure of biological tissues or materials. The contrast of its signal comes from the spatial variation of the optical reflection (scattering) properties inside the biological tissues or materials. It can identify the characteristics of coronary plaques that cannot be accurately identified by angiography or intravascular ultrasound. Although the excellent spatial resolution of OCT can identify many plaque structural features, OCT cannot identify the components of the plaque. NIRS is a new intravascular imaging technology that can provide chemical assessments related to the presence of cholesterol esters in the lipid core and can distinguish cholesterol from collagen in coronary plaques through its unique spectral fingerprint. NIRS can accurately detect lipid-rich plaques in human tissues.

[0056] Existing technologies for detecting plaques based solely on single-modality images are ineffective. Specifically, OCT images alone can identify the structure of different plaques, but cannot accurately distinguish the specific components of the plaques (e.g., fibers, lipids, calcifications, fibrolipids, etc.). Alternatively, near-infrared spectroscopy (NIRS) images alone can be used to observe the distribution of plaques within blood vessels, but the specific location of the plaques cannot be determined. Therefore, hybrid imaging technology based on OCT and NIRS can accurately estimate the structure and composition of lipid plaques within a patient's blood vessels, which has significant positive implications for both physicians and patients.

[0057] In order to solve the above technical problems, the embodiments of the present application provide a lipid plaque assessment system and method based on multimodal intracavitary imaging.

[0058] The technical solution of the present application is described in detail below with reference to the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to be used to explain the present application, but should not be understood as limiting the present application.

[0059] In response to the problems existing in OCT / NIRS hybrid imaging technology, an embodiment of the present application provides a multimodal intracavitary imaging system, which adopts two swept-frequency light sources, one of which emits light with a wavelength length suitable for OCT imaging, and the other swept-frequency light source can emit light with a wavelength length suitable for NIRS to evaluate and analyze lipid plaques; and the two swept-frequency light sources are connected to the imaging probe through a time-division multiplexer. In this way, OCT imaging with higher imaging accuracy and NIRS evaluation results with better absorption peak spectrum can be obtained respectively, which can improve the reproducibility of lipid content evaluation in the corresponding area of ​​the blood vessel wall, thereby providing accurate evaluation results to guide doctors.

[0060] Specifically, refer to Figure 1 As shown, Figure 1 This is a block diagram of the overall structure of the oblique intracavitary imaging system provided by an embodiment of the present application. In one possible implementation, the multimodal intracavitary imaging system provided by the present application includes: a first swept-frequency light source 101, a second swept-frequency light source 102, a first fiber coupler 103, a time division multiplexer 104, and an imaging catheter 105.

[0061] Specifically, in the embodiment of the present application, the first swept-frequency light source 101 and the second swept-frequency light source 102 may be swept-frequency laser light sources, such as superluminescent diodes.

[0062] The first swept-frequency light source 101 can emit light of a wavelength suitable for OCT imaging, such as near-infrared light with a central wavelength of 1310 nm. The second swept-frequency light source 102 can emit light of a wavelength suitable for NIRS assessment imaging, such as near-infrared light with a central wavelength of 1200 nm.

[0063] Specifically, in the embodiment of the present application, the first fiber optic coupler 103 can be a 1x2 type fiber optic coupler, and its splitting ratio can be 90 / 10; wherein, 90 splits of light can be used as sample light to detect the sample to be detected (such as vascular tissue), and 10 splits of light can be used as reference light to be mixed with the detection echo and used as a reference.

[0064] In the embodiment of the present application, the first swept-frequency light source 101 is connected to the first fiber coupler 103; the sample arm of the first fiber coupler 103 is optically connected to the imaging catheter 105 via a time division multiplexer 104; and the second swept-frequency light source 102 is optically connected to the imaging catheter 105 via a time division multiplexer 104.

[0065] Specifically, the pigtail of first swept-frequency light source 101 is connected to first fiber coupler 103. The sample arm of first fiber coupler 103, which can be a 90-degree splitter arm, is connected to time-division multiplexer 104, which is then connected to imaging catheter 105. In this way, light emitted by first swept-frequency light source 101 can pass through first fiber coupler 103, time-division multiplexer 104, and imaging catheter 105 to illuminate vascular tissue, and imaging of the cavity is performed based on the echo signal.

[0066] It should be noted that, in the embodiment of the present application, the pigtail of the second swept-frequency light source 102 is connected to the imaging catheter 105 via the time-division multiplexer 104. In this way, the light emitted by the second swept-frequency light source 102 can be irradiated onto the vascular tissue through the time-division multiplexer 104 and the imaging catheter 105, and the lipid plaque in the blood vessel can be evaluated based on the echo signal.

[0067] It can be understood that the time division multiplexer 104 can allow two or more optical wavelength signals to transmit information through different optical channels simultaneously in the same optical fiber. Therefore, the first swept frequency light source 101 and the second swept frequency light source 102 can be transmitted in the same optical fiber to the imaging catheter 105 after passing through the time division multiplexer 104.

[0068] In an embodiment of the present application, a first swept-frequency light source 101 and a second swept-frequency light source 102 are provided, each emitting light of different central wavelengths. After the light emitted by the first swept-frequency light source 101 is split by a first fiber coupler 103, the sample light passes through a time-division multiplexer 104 and enters an imaging catheter 105 to detect intravascular tissue. Meanwhile, the light emitted by the second swept-frequency light source 102 passes through the time-division multiplexer 104 and enters the imaging catheter 105 to detect intravascular tissue. In this way, the first swept-frequency light source 101 can independently identify the surface microstructure within the blood vessel, while the second swept-frequency light source 102 can independently assess the lipid content in the corresponding region of the vascular wall. Compared to the prior art, when assessing lipid content, light with a central wavelength having a better absorption peak can be used for assessment and identification, thereby improving the reproducibility of imaging for identifying lipid plaques or vulnerable plaques.

[0069] In one embodiment, Figure 1 As shown, the imaging catheter 105 includes two imaging probes 1051, one of the two imaging probes 1051 is connected to the first swept-frequency light source 101 and the second swept-frequency light source 102, respectively.

[0070] The imaging probe 1051 connected to the first swept-frequency light source 101 and the second swept-frequency light source 102 respectively can be used to irradiate the light emitted by the first swept-frequency light source 101 and the second swept-frequency light source 102 onto the intracavitary tissue and receive echo signals; the other of the two imaging probes 1051 is used to receive the echo signal of the second swept-frequency light source 102.

[0071] Specifically, refer to Figure 1 As shown, in an embodiment of the present application, a fiber optic coupler can be provided in the imaging catheter 105. The fiber optic coupler can be of 1x2 type, and the two ports of the fiber optic coupler can be respectively connected to the two imaging probes 1051, so that the light emitted by the first swept-frequency light source 101 and the second swept-frequency light source 102 is irradiated from the imaging probe 1051 onto the vascular tissue 200 and the lipid plaque 300.

[0072] After the near-infrared light is irradiated onto the vascular tissue 200 or the lipid plaque 300 , it is reflected or scattered by the vascular tissue 200 or the lipid plaque 300 . The two imaging probes 1051 can receive, detect or detect the reflected or scattered echo light.

[0073] Specifically, refer to Figure 1 As shown, in the embodiment of the present application, Figure 1 The imaging probe 1051 on the lower side shown in the figure can be connected to the first swept-frequency light source 101 and the second swept-frequency light source 102, respectively. That is, the imaging probe 1051 can irradiate the light emitted by the first swept-frequency light source 101 and the second swept-frequency light source 102 onto the vascular tissue 200 or the lipid plaque 300 in different time periods, and receive the echo light reflected or scattered by the light of the first swept-frequency light source 101 and the second swept-frequency light source 102 after passing through the vascular tissue 200 or the lipid plaque 300.

[0074] It should be noted that, in the embodiments of the present application, only Figure 1 The lower imaging probe 1051 shown in the figure is described as an example. Those skilled in the art will understand that in some optional examples, another imaging probe 1051 can be connected to the first swept-frequency light source 101 and the second swept-frequency light source 102 respectively. The specific position of the imaging probe 1051 is not limited in this embodiment of the application.

[0075] Based on the above embodiment, by controlling the first swept-frequency light source 101 and the second swept-frequency light source 102 to emit light at different time periods, mutual interference between the light emitted by the first swept-frequency light source 101 and the second swept-frequency light source 102 can be avoided. Furthermore, one imaging probe 1051 is used to irradiate the light emitted by the second swept-frequency light source 102 onto the vascular tissue 200 or the lipid plaque 300, while the other imaging probe 1051 is used to receive the echo light from the lipid plaque 300. This prevents mutual interference between the echo light and the incident light, thereby improving the accuracy of the detection imaging and the reproducibility of the assessment results.

[0076] In another possible implementation, the multimodal intracavitary imaging system provided in the embodiment of the present application further includes a rotary joint 106 connected between the time division multiplexer 104 and the imaging catheter 105 , and the echo signals of the two imaging probes 1051 can be transmitted through the optical splitting paths of the rotary joint 106 .

[0077] By way of example and not limitation, rotary joint 106 may be a dual optical path slip ring.

[0078] In one embodiment, Figure 2 The internal structure block diagram of the rotary joint provided by the embodiment of the present application. In the embodiment of the present application, a third optical fiber coupler 1061 is provided in the dual optical path slip ring, and the pigtail of the time division multiplexer 104 is connected to the first port of the rotary joint 106 (specifically, Figure 2 The second port of the rotary joint 106 (specifically, it can be Figure 2 The B port in the figure is connected to the imaging catheter 105; the third port of the rotary joint 106 (specifically, it can be Figure 2 The C port in the image sensor 105 is connected to the control device 109 through the photoelectric detector 108. The control device 109 is used to analyze the echo of the imaging probe 1051 and display the imaging.

[0079] Illustratively, the control device 109 includes a signal processing and display device 1091. The signal processing and display device 1091 may be a liquid crystal display or a display. The liquid crystal display or the display may include a built-in processor capable of analyzing and processing the echo signal, such as a central processing unit (CPU) or a microcontroller unit (MCU). Of course, the processor may also be other types of processors, which are not listed in detail in the embodiments of this application.

[0080] In this way, the signal processing and display device 1091 in the control device 109 can process the echo signal into an image or other data that can be recognized and read by the guiding doctor and display it, so as to facilitate the guiding doctor to perform analysis.

[0081] In the embodiment of the present application, by providing a rotary joint, the echo light of the imaging catheter 105 can be propagated in a split light path, thereby avoiding mutual interference between the echo light of OCT imaging and NIRS evaluation, and improving the accuracy and reproducibility of intravascular imaging.

[0082] It is understandable that in order to avoid the echo light from interfering with the light emitted by the first swept-frequency light source 101 or the second swept-frequency light source 102. Figure 1As shown, in the embodiment of the present application, a first optical fiber circulator 107 is connected between the time division multiplexer 104 and the rotary joint 106, one port of the first optical fiber circulator 107 is connected to the time division multiplexer 104, and the second port of the first optical fiber circulator 107 is connected to the rotary joint 106;

[0083] The echo of the imaging probe 1051 connected to the first swept-frequency light source 101 can enter the first optical fiber circulator 107 through the second port of the first optical fiber circulator 107 and be transmitted outward from the third port of the first optical fiber circulator 107 .

[0084] In this way, the incident light can only propagate in one direction within the first fiber optic circulator 107, that is, the light entering from port one can only be emitted from port two, and the light entering from port two can only be emitted from port three; this can effectively avoid interference between the light emitted by the first swept-frequency light source 101 and the echo light of the imaging probe 1051.

[0085] It should be noted that, in the embodiment of the present application, the second port of the first optical fiber circulator 107 is connected to the first port of the rotary joint 106 .

[0086] Optionally, in an embodiment of the present application, the reference arm of the first fiber optic coupler 103 is connected to the collimator 111 through the second fiber optic circulator 110. The light emitted by the collimator 111 can be reflected by the mirror 112 and transmitted outward from the third port of the second fiber optic circulator 110 through the second port of the second fiber optic circulator 110.

[0087] Specifically, the coherent wavelength length of the OCT optical system can be adapted by adjusting the position of the mirror 112 .

[0088] Optionally, the three ports of the first fiber circulator 107 and the three ports of the second fiber circulator 110 are connected to a balanced photodetector 114 through a second fiber coupler 113 . The balanced photodetector 114 is used to convert the optical signal output by the second fiber coupler 113 into an electrical signal.

[0089] The second optical fiber coupler 113 may be a 2x2 coupler with a splitting ratio of 50 / 50.

[0090] That is, the echo light from the first swept-frequency light source 101 and the echo light reflected by the mirror interfere with each other in the second fiber coupler 113, and then are divided into two beat wave signals with a phase difference of π / 2 and enter the balanced photodetector 114. The balanced photodetector 114 converts the optical signal into an electrical signal to facilitate subsequent analysis and processing.

[0091] In the embodiment of the present application, by providing a balanced photodetector, the influence of receiver noise and electronic circuit noise on weak light signal detection can be significantly eliminated, thereby improving the accuracy of OCT imaging.

[0092] Specifically, the balanced photodetector 114 is connected to the signal processing and display device 1091 in the control device 109 . The signal processing and display device 1091 is used to analyze the electrical signal output by the balanced photodetector 114 and display the image.

[0093] In other possible implementations, the control device 109 further includes a control device 1092, which is respectively connected to the first swept-frequency light source 101 and the second swept-frequency light source 102, and the control device 1092 can respectively control the first swept-frequency light source 101 and the second swept-frequency light source 102 to start alternately.

[0094] Specifically, in the embodiment of the present application, the first swept-frequency light source 101 and the second swept-frequency light source 102 can operate in time periods in an electrically controlled manner through the control device 1092, that is, drive OCT imaging and NIRS evaluation respectively.

[0095] For example, the control device 1092 may be a computer, a laptop, a tablet computer, or a personal digital computer with a controller.

[0096] The following describes in detail the optical path transmission of the multi-modal intracavity imaging system provided in the embodiment of the present application:

[0097] After the light emitted by the first swept-frequency light source 101 passes through the first fiber coupler 103, the light beam is split into two, one as a sample signal (splitting light 90) and the other as a reference signal (splitting light 10). The sample signal passes through the time division multiplexer 104 and enters one port of the first fiber circulator 107. The light signal exits through the second port of the first fiber circulator 107 and enters the rotary joint 106. The third fiber coupler 1061 inside the rotary joint 106 guides the signal light into the imaging catheter 105. The signal light hits the target tissue (such as the blood vessel wall 200 or lipid plaque 300) through the imaging probe 1051 of the imaging catheter 105. The echo signal is coupled into the optical fiber through the imaging probe 1051, enters the second port of the first fiber circulator 107 through the first port of the rotary joint 106, and exits from the third port of the first fiber circulator 107 and enters the second fiber coupler 113.

[0098] The reference signal is connected to port one of the second fiber circulator 110. From port two of the second fiber circulator 110, it enters fiber collimator 111 and emerges as spatial light. It then strikes mirror 111, where the echo from mirror 111 is collected and coupled into the optical fiber by fiber collimator 111. The echo passes through port two of the second fiber circulator 110 and emerges from port three, entering second fiber coupler 113. The two light beams undergo coherent interference in second fiber coupler 113, splitting into two beat wave signals with a phase offset of π / 2 and passing into balanced photodetector 114.

[0099] The light output from the second swept-frequency light source 102 is connected to the time division multiplexer 104 via a pigtail, and is emitted into one port of the first optical fiber circulator 107. It is emitted from the second port of the first optical fiber circulator 107 and enters the first port of the rotary joint 106. The light beam is emitted from the second port of the rotary joint 106 and enters the imaging catheter 105. After passing through the imaging probe 1051 of the imaging catheter 105, it hits the target tissue (blood vessel wall 200 or lipid plaque 300, etc.). After the signal is loaded with the information of the lipid plaque 300, it is coupled into the optical fiber through the imaging probe 1051 of the imaging catheter 105. The optical signal is output from the second port of the rotary joint 106 and its third port and enters the photodetector 108.

[0100] In order to accurately assess the lipid content of lipid plaques in patients’ blood vessels, the present application also provides a multimodal data fusion method. Figure 1 The control device 1092 of the multimodal intracavitary imaging system shown in the figure controls the first swept-frequency light source 101 and the second swept-frequency light source 102 to start alternately to detect the intracavitary tissue. The imaging probe 1051 irradiates the light emitted by the first swept-frequency light source 101 and the second swept-frequency light source 102 onto the intracavitary tissue, and transmits the echo signal to the signal processing and display device 1091, which can analyze the echo signal and display the imaging. After acquiring the image, the control device 109 can fuse the acquired image data through the multimodal data fusion method provided in this application, thereby accurately evaluating the lipid content of the intracavitary tissue.

[0101] In the embodiment of the present application, if the first swept-frequency light source 101 emits light of a wavelength suitable for OCT imaging, and the second swept-frequency light source 102 emits light of a wavelength suitable for NIRS imaging, then when the control device 1092 controls the first swept-frequency light source 101 and the second swept-frequency light source 102 to alternately start detecting the intracavitary tissue, the signal processing and display device 1091 can obtain the OCT image of the intracavitary tissue and the NIRS image corresponding to the OCT image. Figure 3 Based on Figure 1 The system shown acquires an image of the intracavitary tissue, wherein: Figure 3(a) is an OCT image acquired by the signal processing and display device 1091. The OCT image can show the structure of lipid plaques in the intraluminal tissue. Figure 3 (b) is a combined image containing OCT images and NIRS images. Figure 3 (b) It can be seen that the pixel value of the pixel point in the axial direction of the NIRS image represents the absorption value of the light wave by the lipid plaque in the OCT image in the axial direction.

[0102] The flowchart of the multimodal data fusion method provided in this application is as follows Figure 4 The method includes:

[0103] S401, obtain Figure 1 The multimodal intracavitary imaging system shown in FIG2 acquires an OCT image sequence and a NIRS image sequence of intracavitary tissue, wherein the OCT image sequence includes multiple OCT images, and the NIRS image sequence includes multiple NIRS images corresponding one-to-one to the multiple OCT images.

[0104] It is understandable that the control device 109 in the multimodal intracavitary imaging system can control the first swept-frequency light source 101 and the second swept-frequency light source 102 to alternately start detecting different positions of the intracavitary tissue. The imaging probe 1051 irradiates the light emitted by the first swept-frequency light source 101 onto the intracavitary tissue, and receives the avoidance signal to transmit the echo signal to the control device 109. The control device 109 analyzes the echo signal to obtain an OCT image. The imaging probe 1051 can irradiate the light emitted by the second swept-frequency light source 102 onto the intracavitary tissue, and another imaging probe 1051 receives the echo signal to transmit the echo signal to the control device 109. The control device 109 analyzes the echo signal to obtain multiple NIRS images corresponding to the above-mentioned multiple OCT images.

[0105] It should be noted that the image sequence may be multiple images of the same blood vessel lumen acquired consecutively by the multimodal intracavitary imaging system, or may be multiple consecutive frames of images in a video acquired by the multimodal intracavitary imaging system.

[0106] Among them, different axial directions in the NIRS image show different degrees of near-infrared attenuation. Figure 3 As shown in (b), each axis in the NIRS image reflects the attenuation degree of near-infrared light by the intracavitary tissue through a normalized value between 0 and 1, and finally presents the attenuation spectrum of the intracavitary cross-section, which is used as a parameter to reflect the lipid distribution in the intracavitary cross-section.

[0107] S402: Register each OCT image with the NIRS image corresponding to the OCT image.

[0108] It should be noted that when using Figure 1 When the system shown captures OCT and NIRS images, there is a certain angle between the OCT and NIRS data acquisition probes, making it impossible to directly obtain perfectly aligned OCT and NIRS images for subsequent data fusion analysis. Therefore, before fusing each OCT image with its corresponding NIRS image, each OCT image must be registered with its corresponding NIRS image to facilitate subsequent operations. The specific implementation method is as follows:

[0109] In one possible implementation, for each OCT image and the NIRS image corresponding to the OCT image, the position of the guidewire in the OCT image and the NIRS image can be identified to determine the direction and angle of relative rotation between the OCT image and the NIRS image, thereby aligning the OCT image and its corresponding NIRS image.

[0110] In one embodiment, for each OCT image, the guidewire in the OCT image can be identified by the target detection model. Exemplarily, the target detection model can be a YOLO network. The YOLO network includes 24 convolutional layers and 2 fully connected layers. First, the input OCT image is divided into blocks, that is, the input image of size 224*224 is divided into 7*7 cells, and the size of each cell is 36*36. The YOLO network can mark a preset number of bounding boxes in the cells containing the guidewire. In the process of identifying the guidewire, the parameters that need to be determined include the bounding box (x, y, w, h), confidence and category probability (category is 1). Where (x, y) is the center coordinate of the bounding box, and (w, h) represents the width and height of the bounding box. The confidence is defined as If there is an object in the cell, Pr(Object)=1, otherwise Pr(Object)=0; It represents the ratio of the overlapping area between the predicted box and the true box to the total area of ​​the predicted box and the true box.

[0111] For each NIRS image corresponding to each OCT image, the largest continuous region with an axial value of zero in the NIRS image is calculated as the guidewire position in the NIRS image. It is understood that guidewires are generally made of metal. In the NIRS image, the axial value at the guidewire location is zero, so the largest continuous region with an axial value of zero in the NIRS image is the guidewire position.

[0112] After obtaining the position of the guide wire in each OCT image and the NIRS image corresponding to the OCT image based on the above embodiment, the pixel matrix of the OCT image is rotated relative to the pixel matrix of the NIRS image according to the offset angle of the guide wire position in the two images, so that the OCT image and the NIRS image are aligned.

[0113] S403 , identifying the lipid plaque region in each OCT image, and determining a first judgment value for each OCT image according to the area of ​​the lipid plaque region in the plurality of OCT images.

[0114] In one possible implementation, this application combines semantic segmentation methods with traditional digital image processing methods based on pixel texture features, statistical values, etc. to accurately identify lipid plaque areas in each OCT image. The specific implementation is as follows:

[0115] Step 1: Input each OCT image into the trained U-Net network to obtain the first lipid segmentation mask of each OCT image.

[0116] Specifically, the U-Net network includes an encoder and a decoder. Among them, the encoder includes a plurality of convolutional layer modules, each of which includes two convolutional layers, a RELU activation layer and a maximum pooling layer for extracting features and performing downsampling. The decoder includes a plurality of deconvolution modules, each of which includes a deconvolution layer and a RELU activation layer. The final network output is a feature map of two channels, representing two categories, namely background and foreground. For the OCT image in the embodiment of the present application, the foreground is the lipid plaque area, and the background is the non-lipid plaque area. The feature maps of the two channels are then input into the softmax function to output the lipid segmentation mask image.

[0117] In an embodiment of the present application, each OCT image is input into the U-Net network respectively. The U-Net network can identify whether each pixel in the OCT image belongs to the lipid plaque area, thereby obtaining the first lipid plaque mask of each OCT image.

[0118] Among them, the size of the first lipid plaque mask is the same as the size of the corresponding OCT image. The pixel values ​​in the first lipid segmentation mask and the second lipid segmentation mask represent the judgment results of the pixel points at the same position in the OCT image, and the judgment result is whether the pixel points are in the lipid plaque area. Exemplarily, the pixel values ​​of the pixels in the first lipid plaque mask are 0 or 1. If the pixel value of the pixel point in the first lipid plaque mask is 0, it means that the pixel point at the same position in the OCT image does not belong to the lipid plaque area; if the pixel value of the pixel point in the first lipid segmentation mask is 1, it means that the pixel point at the same position in the OCT image belongs to the lipid area.

[0119] Step 2: Obtain a multidimensional feature vector for each OCT image and input the multidimensional feature vector into the trained random forest to obtain a second lipid segmentation mask for each OCT image.

[0120] In this embodiment, features of each OCT image can be extracted using traditional digital image processing methods, and the multidimensional feature vector can be input into a trained random forest to determine the category of each pixel point, thereby obtaining a second lipid segmentation mask for each OCT image.

[0121] In one example, a multidimensional feature vector of an OCT image can be obtained based on a feature extraction method of a gray-level co-occurrence matrix (such as ASM energy (Angular Second Moment), inertia, contrast, entropy, autocorrelation, etc.). For example, it is assumed that the multidimensional feature vector of an OCT image is obtained based on the three features of ASM energy, inertia, and contrast in the gray-level co-occurrence matrix. For each pixel in the OCT image, the ASM energy value a1, inertia value a2, and contrast value a3 of the pixel can be calculated based on the correlation between the pixel itself and its neighboring pixels. Therefore, a feature vector A=[a1, a2, a3] containing these three features will be extracted for each pixel. Assuming that the size of an OCT image is M×N, the size of the multidimensional feature vector of the OCT image is M×N×3. The multidimensional feature vector of the OCT image represents the edge texture information of the OCT image.

[0122] In another example, a multidimensional feature vector of an OCT image can be obtained based on a feature extraction method using a transform domain and a filter. For example, the OCT image can be first converted to a transform domain using a linear transformation method (e.g., discrete cosine transform, local Fourier transform, Walsh-Hadamard transform, wavelet transform, etc.) to obtain multiple feature images; then, the multiple feature images are processed using a filter (e.g., a bandpass filter, a notch filter, etc.) to obtain a multidimensional feature vector of the OCT image.

[0123] The multidimensional feature vector of the OCT image obtained above is input into the trained random forest. The random forest can classify each pixel in the OCT image according to the multidimensional feature vector of the OCT image, and finally obtain the second lipid segmentation mask.

[0124] Based on the above embodiment, the lipid plaque area in the OCT image is relatively smooth, and the boundary of the lipid plaque is relatively blurred. Based on the first-order, second-order or high-order statistical characteristics of the grayscale of the pixel and its neighborhood, the autocorrelation function and other features can effectively extract the texture features of the local serially repeated and relatively smooth areas in the image, which can improve the classification accuracy and thus improve the accuracy of the mask.

[0125] Step three: determining the lipid plaque area in each OCT image according to the first lipid segmentation mask and the second lipid segmentation mask of each OCT image.

[0126] In one embodiment, for each OCT image, if the pixel value of a pixel in the first lipid segmentation mask and the pixel value of a pixel at the same position in the second lipid segmentation mask both indicate that the pixel belongs to the lipid plaque region, then the pixel at the same position in the OCT image is considered to be a lipid plaque region. In other words, the first lipid segmentation mask and the second lipid segmentation mask are ANDed together, and for each pixel in the OCT image, only if the discrimination result based on the U-Net network is the same as the discrimination result based on traditional digital image processing methods can the pixel be determined to be a lipid plaque.

[0127] It should be noted that the embodiment of the present application can also obtain multiple lipid segmentation masks of the OCT image through two or more methods, and then perform an AND operation on the multiple lipid segmentation masks to identify the lipid plaque area in the OCT image.

[0128] Step 4: Obtain the area of ​​the lipid plaque region in multiple OCT images, divide the area of ​​the lipid plaque region in each OCT image by the maximum value of the areas of the lipid plaque regions in multiple OCT images, and obtain a first judgment value for each OCT image.

[0129] In one embodiment, in an OCT image sequence, the number of pixels belonging to lipid plaques in each OCT image is counted. The lipid plaque area of ​​each OCT image is the total number of pixels belonging to lipid plaques in the OCT image multiplied by the pixel resolution of the OCT image. Assume that the OCT image sequence includes H OCT images, among which the lipid plaque area of ​​the hth OCT image is the largest, and the largest area is S h Then the first judgment value x1 of any OCT image in the OCT image sequence can be expressed by formula (1) as:

[0130]

[0131] Where k represents the kth OCT image in a sequence of multiple OCT images, S k represents the lipid plaque area of ​​the kth OCT image. The first judgment value of the entire OCT image sequence can be expressed as

[0132] S404 , determining a second judgment value for each NIRS image according to the sampling frequency used when acquiring the NIRS image sequence and the number of samples of each NIRS image, where the number of samples of the NIRS image is the number of absorption values ​​greater than a threshold.

[0133] In one embodiment, for each NIRS image, the numerical value of the NIRS image in each axial direction is expressed as the absorption value of the wavelength by the intracavitary tissue at that position. The maximum pixel value in the NIRS image is obtained, and each pixel value in the NIRS image is divided by the maximum pixel value to obtain a normalized NIRS image. The pixel values ​​in the normalized NIRS image are between 0 and 1, where "1" and "0" correspond to the maximum and minimum values ​​absorbed by the intracavitary tissue, respectively. When the normalized absorption value in a certain axial direction is greater than 0.6, the tissue can be considered to be lipid. Assuming that the acquisition frequency of each NIRS image in the NIRS image sequence is the same, the sampling frequency is N (that is, N samples are collected in the entire axial direction of each NIRS image), and the number of samples n with absorption values ​​greater than 0.6 in these N samples is counted, then the second judgment value x2 of the NIRS image can be expressed as:

[0134]

[0135] Wherein, n<=N; k represents the kth NIRS image in the NIRS image sequence; The judgment value of the entire NIRS image sequence can be expressed as The number of NIRS images in the NIRS image sequence is the same as the number of OCT images in the OCT image sequence.

[0136] S405 , fusing the first judgment value of each OCT image and the second judgment value of the NIRS image corresponding to each OCT image to determine the lipid level indicated by each OCT image.

[0137] In order to accurately obtain the distribution of intraluminal lipid plaques from a three-dimensional perspective, it is necessary to perform data fusion on each OCT image and the NIRS image corresponding to each OCT image one by one, so as to obtain the distribution of lipid plaques in the longitudinal direction of the blood vessels and evaluate the lipid content of the lipid plaques.

[0138] In one possible implementation, for each OCT image, a support matrix is ​​determined based on the first judgment value of the OCT image and the second judgment value of the NIRS image corresponding to the OCT image. A weight coefficient is obtained based on the eigenvector corresponding to the maximum eigenvalue of the support matrix. Then, a weighted sum of the first judgment value of the OCT image and the second judgment value of the NIRS image corresponding to the OCT image is performed according to the weight coefficient to obtain a fusion value. The fusion value is input into a Kalman filter to obtain an evaluation parameter for each OCT image, and the evaluation parameter is used to characterize the lipid level. The specific implementation method is as follows:

[0139] First, for each OCT image, the support matrix is ​​determined based on the first decision value x1 of the OCT image and the second decision value x2 of the NIRS image corresponding to the OCT image. The support matrix R can be expressed as:

[0140]

[0141]

[0142] d ij =|x i -x j | (5)

[0143] In the above formula, i = 1, 2; j = 1, 2. When fusion is performed, we need to find the weight coefficient of each data in the two judgment values. Expressed in matrix form as in, V=[v1 v2] T represents the eigenvector corresponding to the maximum eigenvalue λ of the matrix R, then the i-th data x i The weight coefficient can be expressed as formula (6):

[0144]

[0145] The normalized value of the OCT image and the NIRS judgment value are weighted fused, and the fusion result (fusion value) is:

[0146] Finally, considering the consistent correlation between each image in the image sequence and its previous and next image data, the above fusion value is input into the Kalman filter to obtain the evaluation parameter x(k) of the lipid plaque of each image, which represents the lipid degree of the lipid plaque:

[0147]

[0148] It can be understood that, by evaluating the lipid plaques in each frame of OCT image in the long axis direction of the OCT image sequence, a continuous lipid evaluation result in the long axis direction of the blood vessel can be obtained. Figure 5 This example provides a multimodal data fusion result graph. Based on the lipid assessment results, a waveform graph of intraluminal lipid levels can be generated. Peaks in this waveform indicate the highest lipid levels in the corresponding lipid plaque within the vascular lumen. The multimodal lipid plaque assessment results provided by this application are more accurate than those obtained by traditional methods.

[0149] In one embodiment, the present application provides a training method for a YOLO network. The specific training process is as follows:

[0150] Step 1: Obtain training samples containing guidewires.

[0151] Specifically, the training samples include OCT image samples and NIRS image samples. OCT image samples containing guidewires are manually selected and the guidewires in the OCT image samples are manually labeled. To obtain more training samples, data augmentation is required to increase the number of OCT image samples.

[0152] For example, the number of OCT image samples marked with guidewires is expanded by rotating, adjusting contrast, and / or adding noise (such as random noise, salt and pepper noise, etc.), so as to obtain a sufficient number of training samples containing guidewires.

[0153] The NIRS image sample corresponding to each OCT image sample needs to be rotated by the same angle for subsequent registration.

[0154] Step 2: Input the training samples into the initial YOLO network for iterative training.

[0155] During the training process, considering that the guidewire occupies a relatively small portion of the pixels in the entire image, the weights of the positioning error and the classification error should not be equal. Therefore, when selecting the loss function, a weighted loss function that is more suitable for small target recognition should be selected. For example, the loss weight of the bounding box coordinate prediction is increased, and the loss weight of the confidence prediction of the bounding box that does not contain the target is reduced. Specifically, the loss function of the YOLO network is formula (8):

[0156]

[0157] In the above formula, The training samples are input into the initial YOLO network for iterative training. When the loss function meets the preset requirements, it means that the model has converged, that is, the initial YOLO network has completed training, and a trained YOLO network is obtained.

[0158] In another embodiment, the present application provides a method for training a U-net network model, the specific process is as follows:

[0159] Step 1: Obtain an OCT image sample with lipid plaque areas marked.

[0160] In one possible implementation, OCT image samples containing lipid plaques are manually selected from the OCT pullback data. The lipid plaque regions in these OCT image samples are then manually annotated. To obtain more training samples, data augmentation is employed to increase the number of OCT image samples. Exemplarily, the annotated OCT image samples are augmented by rotating them, adjusting their contrast, and / or adding noise (e.g., random noise, salt and pepper noise, etc.).

[0161] Step 2: Input the OCT image samples into the initial U-net network for network training.

[0162] Specifically, an OCT image sample is input into the U-net encoder, which extracts feature information from the salient regions of the sample, generating a multi-channel feature image. The decoder then expands the size of the feature image, ultimately outputting a two-channel feature map. These two feature maps are then used as input to the softmax function, which calculates the softmax category with the highest probability. Finally, backpropagation training is performed using the cross-entropy loss function. When the loss function meets the preset conditions, the model has converged, indicating that the initial U-net network has completed training, resulting in a trained U-net network model.

[0163] Based on the multimodal data fusion method provided by the present application, the size and structure of the plaque in the blood vessel are determined by identifying the lipid plaque area contained in the OCT image, and the first judgment value of each OCT image is determined based on the area of ​​the lipid plaque area of ​​the multiple OCT images. At the same time, based on the absorption value of the spectrum by the tissue in the blood vessel displayed in the NIRS image, it is determined whether the tissue in the blood vessel is lipid, thereby determining the second judgment value of each NIRS image based on the sampling frequency and the number of samples of each NIRS image. By fusing the first judgment value and the second judgment value, the lipid level indicated by each of the OCT images can be determined. Compared with the existing method of evaluating lipid plaques or vulnerable plaques based on single-modality images, the present application utilizes the complementarity of OCT images and NIRS images to fuse image data of multiple modalities, thereby evaluating the lipid level of lipid plaques and improving the accuracy of the evaluation.

[0164] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0165] An embodiment of the present application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein when the processor executes the computer program, the steps of the multimodal data fusion method described in the above embodiment are implemented.

[0166] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the steps in the multimodal data fusion method described in the above embodiment.

[0167] An embodiment of the present application provides a computer program product. When the computer program product runs on a cleaning robot, the cleaning robot can implement the steps of the multimodal data fusion method described in the above embodiment when executing the computer program product.

[0168] References to "one embodiment" or "an example" in this application mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the application. Thus, phrases such as "in one embodiment," "in one example," "in one possible implementation," and "in another possible implementation" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0169] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one of such features. It should also be understood that the term "and / or" used in this specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, including and including such combinations.

[0170] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A multimodal intracavity imaging system, characterized in that: include: A first swept-frequency light source (101), a second swept-frequency light source (102), a first optical fiber coupler (103), a time division multiplexer (104), an imaging catheter (105), and a control device (109); The first swept-frequency light source (101) is connected to the first optical fiber coupler (103); the sample arm of the first optical fiber coupler (103) is connected to the imaging catheter (105) optical signal via the time division multiplexer (104); the second swept-frequency light source (102) is connected to the imaging catheter (105) optical signal via the time division multiplexer (104); The first swept-frequency light source (101) and the second swept-frequency light source (102) can emit light of different central wavelengths; the imaging catheter (105) can detect and image the intracavitary tissue based on the light emitted by the first swept-frequency light source (101) and the second swept-frequency light source (102); The control device (109) is used to receive the echo signals corresponding to the first swept-frequency light source (101) and the second swept-frequency light source (102) through the imaging catheter (105), and to analyze the echo signals corresponding to the first swept-frequency light source (101) to obtain a plurality of OCT images, and to analyze the echo signals corresponding to the second swept-frequency light source (102) to obtain a plurality of NIRS images, wherein the plurality of NIRS images correspond one-to-one to the plurality of OCT images; The control device (109) is further used to: register each of the OCT images with the NIRS image corresponding to the OCT image; identify the lipid plaque area in each of the OCT images, and determine a first judgment value for each of the OCT images based on the area of ​​the lipid plaque area in the plurality of OCT images; The second judgment value of each NIRS image is determined according to the sampling frequency used when acquiring the NIRS image and the number of samples of each NIRS image, where the number of samples of the NIRS image is the number of absorption values ​​greater than a threshold value; the first judgment value of each OCT image and the second judgment value of the NIRS image corresponding to each OCT image are fused to determine the lipid level indicated by each OCT image.

2. The multimodal intracavity imaging system according to claim 1, wherein: The imaging catheter (105) comprises two imaging probes (1051), one of the two imaging probes (1051) being connected to the first swept-frequency light source (101) and the second swept-frequency light source (102), respectively; The imaging probe (1051) connected to the first swept-frequency light source (101) and the second swept-frequency light source (102) respectively can be used to irradiate the light emitted by the first swept-frequency light source (101) and the second swept-frequency light source (102) onto the intracavitary tissue and receive echo signals; the other of the two imaging probes (1051) is used to receive the echo signal of the second swept-frequency light source (102).

3. The multimodal intracavity imaging system according to claim 2, wherein: The multimodal intracavitary imaging system further comprises a rotary joint (106) connected between the time division multiplexer (104) and the imaging catheter (105), and the echo signals of the two imaging probes (1051) can be transmitted through the optical splitting paths of the rotary joint (106).

4. The multimodal intracavity imaging system according to claim 3, characterized in that: The multimodal intracavity imaging system further comprises a first optical fiber circulator (107) connected between the time division multiplexer (104) and the rotary joint (106), wherein one port of the first optical fiber circulator (107) is connected to the time division multiplexer (104), two ports of the first optical fiber circulator (107) are connected to the first port of the rotary joint (106), and the second port of the rotary joint (106) is connected to the imaging catheter (105); The echo of the imaging probe (1051) connected to the first frequency-sweeping light source (101) can enter the first optical fiber circulator (107) through the second port of the first optical fiber circulator (107) and be transmitted outward from the third port of the first optical fiber circulator (107).

5. The multimodal intracavity imaging system according to claim 4, characterized in that: The multi-modal intracavity imaging system further includes a second optical fiber circulator (110), a collimator (111), a mirror (112), a second optical fiber coupler (113), and a balanced photodetector (114); The reference arm of the first optical fiber coupler (103) is connected to the collimator (111) through the second optical fiber circulator (110); the light emitted by the collimator (111) can be reflected by the mirror (112) and transmitted outward from the third port of the second optical fiber circulator (110) through the second port of the second optical fiber circulator (110); The three ports of the first optical fiber circulator (107) and the three ports of the second optical fiber circulator (110) are connected to the balanced photodetector (114) via the second optical fiber coupler (113); the balanced photodetector (114) is used to convert the optical signal output by the second optical fiber coupler (113) into an electrical signal.

6. The multimodal intracavity imaging system according to claim 5, characterized in that: The control device (109) is connected to the balanced photodetector (114) and is connected to the third port of the rotary joint (106) via the photodetector (108). The control device (109) is used to analyze and display the electrical signal output by the balanced photodetector (114) and to analyze and display the echo of the imaging probe (1051). The control device (109) is connected to the first swept-frequency light source (101) and the second swept-frequency light source (102) respectively, and the control device (109) controls the first swept-frequency light source (101) and the second swept-frequency light source (102) to start alternately.

7. A multimodal data fusion method, characterized in that: include: Acquire an OCT image sequence and a NIRS image sequence of intracavitary tissue acquired by the multimodal intracavitary imaging system according to any one of claims 1 to 6, wherein the OCT image sequence includes a plurality of OCT images, and the NIRS image sequence includes a plurality of NIRS images corresponding one-to-one to the plurality of OCT images; Registering each of the OCT images with the NIRS image corresponding to the OCT image; identifying a lipid plaque region in each of the OCT images, and determining a first judgment value for each of the OCT images based on the area of ​​the lipid plaque region in the plurality of OCT images; determining a second judgment value for each of the NIRS images according to a sampling frequency used when acquiring the NIRS image sequence and a sample number of each of the NIRS images, wherein the sample number of the NIRS images is the number of absorption values ​​greater than a threshold; The first judgment value of each of the OCT images and the second judgment value of the NIRS image corresponding to each of the OCT images are fused to determine the lipid level indicated by each of the OCT images.

8. The method according to claim 7, characterized in that The registering each of the OCT images with the NIRS image corresponding to the OCT image comprises: For each of the OCT images and the NIRS image corresponding to the OCT image, the position of the guide wire in the OCT image and the NIRS image is identified respectively, and according to the offset angle of the guide wire position, the rotation angle of the pixel matrix of the OCT image relative to the pixel matrix of the NIRS image is adjusted to align the OCT image with the NIRS image.

9. The method according to claim 7, characterized in that The identifying of lipid plaque areas in each of the OCT images comprises: Inputting each of the OCT images into a trained U-Net network to obtain a first lipid segmentation mask for each of the OCT images; Obtaining a multidimensional feature vector of each of the OCT images, and inputting the multidimensional feature vector into a trained random forest to obtain a second lipid segmentation mask for each of the OCT images; The lipid plaque area in each of the OCT images is determined according to the first lipid segmentation mask and the second lipid segmentation mask of each of the OCT images.

10. The method according to claim 7, characterized in that Determining the first judgment value of each of the OCT images according to the area of ​​the lipid plaque region of the plurality of OCT images includes: Acquiring the area of ​​lipid plaque regions in the plurality of OCT images; The area of ​​the lipid plaque region in each of the OCT images is divided by the maximum value of the areas of the lipid plaque regions in the multiple OCT images to obtain a first judgment value for each of the OCT images.

11. The method according to claim 7, characterized in that Determining the second determination value of each NIRS image according to the sampling frequency used when acquiring the NIRS image sequence and the number of samples of each NIRS image includes: For each of the NIRS images, the number of samples of the NIRS image is divided by the sampling frequency used when acquiring the NIRS image sequence to obtain a second determination value for each of the NIRS images.

12. The method according to any one of claims 7 to 11, characterized in that The step of fusing the first determination value of each of the OCT images with the second determination value of the NIRS image corresponding to each of the OCT images to determine the lipid level indicated by each of the OCT images comprises: For each of the OCT images, determining a support matrix according to a first determination value of the OCT image and a second determination value of the NIRS image corresponding to the OCT image; Obtaining a weight coefficient according to the eigenvector corresponding to the maximum eigenvalue of the support matrix; performing weighted summation on the first judgment value of the OCT image and the second judgment value of the NIRS image corresponding to the OCT image according to the weight coefficient to obtain a fusion value; The fusion value is input into a Kalman filter to obtain an evaluation parameter of each of the OCT images, and the evaluation parameter is used to characterize the lipid level.

Images