Image processing method, device, electronic device and storage medium
Through the registration and light attenuation coefficient analysis of OCT images and angiographic images, the plaque attenuation index is calculated and marked, which solves the problem of insufficient recognition ability of vulnerable plaques in OCT technology, and improves the display effect and recognition efficiency of the image.
Patent Information
- Application Number
- CN202110790290.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-13
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-07-13
AI Technical Summary
The existing optical coherence tomography (OCT) technology has weak ability to recognize vulnerable plaques, and the display effect of OCT images is poor, making it difficult to assist medical staff in judging the load status of vulnerable plaques, resulting in high recognition difficulty and high cost.
By acquiring the registration parameters of OCT images and angiographic images, an optical attenuation coefficient image is generated, and the plaque attenuation index (IPA) value is calculated, and the IPA value is marked on the angiographic images to improve the recognition and display of vulnerable plaques. Intuitive.
It significantly improves the ability to identify vulnerable plaques, allowing medical staff to observe the location and condition of vulnerable plaques more clearly and intuitively, reducing the difficulty and cost of identification.
Smart Images

Figure CN113469986B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of medical image processing technology, and in particular, relates to an image processing method, device, electronic device and storage medium. Background Art
[0002] Optical coherence tomography (OCT) is an imaging technology. It uses the basic principle of weak coherent light interferometer to split the light emitted by the light source into two beams, one of which is emitted to the tissue to be tested, also called the sample arm, and the other is emitted to the reference reflector, also called the reference arm. Then, the two light signals reflected from the tissue to be tested and from the reference reflector are superimposed and interfered. Finally, the light signals are displayed with different intensity grayscales depending on the tissue to be tested, thereby imaging the inside of the tissue.
[0003] The existing traditional optical coherence imaging technology has a weak ability to identify vulnerable plaques. It is difficult to improve the recognition capability from the system and equipment end, and it will also incur high costs. In addition, the display effect of existing OCT images is poor, which is not convenient for assisting medical staff to judge the load of vulnerable plaques. Summary of the invention
[0004] The embodiments of the present application provide an image processing method, device, electronic device, and storage medium, which can solve at least part of the above problems.
[0005] In a first aspect, an embodiment of the present application provides an image processing method, comprising:
[0006] In the process of acquiring an optical coherence tomography (OCT) image group for the object under investigation, acquiring an angiography image group for the object under investigation;
[0007] Registering the OCT image group and each angiography image in the angiography image group to obtain a registration parameter;
[0008] Based on each OCT image in the OCT image group, generating a plaque attenuation coefficient light attenuation coefficient image group corresponding to each OCT image;
[0009] Calculating the IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group;
[0010] Based on the registration parameters, each angiography image in the angiography image group is marked using the IPA value to obtain a target angiography image group.
[0011] It should be understood that by registering the OCT image group and each angiography image in the angiography image group, the registration parameters of the OCT image group and each angiography image, that is, the corresponding relationship, are obtained, so that the light attenuation coefficient image obtained using each OCT image has a consistent corresponding relationship with each angiography image. On this basis, the IPA value obtained based on the light attenuation coefficient image is marked on each angiography image, so that the position and compliance of the vulnerable plaque can be clearly and intuitively observed, thereby improving the ability to identify vulnerable plaques of the explored object.
[0012] In a second aspect, an embodiment of the present application provides an image processing device, including:
[0013] An image acquisition module, used for acquiring a group of angiography images for the object under investigation during the process of acquiring a group of optical coherence tomography (OCT) images for the object under investigation;
[0014] An image registration module, used for registering the OCT image group and each angiography image in the angiography image group to obtain registration parameters;
[0015] An optical attenuation coefficient image generation module, used to generate a plaque attenuation coefficient optical attenuation coefficient image group corresponding to each OCT image based on each OCT image in the OCT image group;
[0016] An IPA value generating module, used for calculating the IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group;
[0017] The image marking module marks each angiography image in the angiography image group by using the IPA value based on the registration parameters to obtain a target angiography image group.
[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method steps described in the first aspect when executed by the processor.
[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, including: the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in the first aspect are implemented.
[0020] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the method steps described in the first aspect above.
[0021] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0023] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present application;
[0024] Figure 2 is a schematic diagram of the correspondence between an OCT image and an angiography image provided in an embodiment of the present application;
[0025] Figure 3 is a schematic diagram of image processing provided by an embodiment of the present application;
[0026] Figure 4 is a flowchart of an image processing method provided by another embodiment of the present application;
[0027] Figure 5 is a flowchart of an image processing method provided by another embodiment of the present application;
[0028] Figure 6 is a schematic diagram of image registration provided by an embodiment of the present application;
[0029] Figure 7 This is a schematic diagram of a U-shaped attention network detecting a pull-back blood vessel provided by an embodiment of the present application;
[0030] Figure 8 This is a schematic diagram of a cyclic generative adversarial network provided by an embodiment of the present application;
[0031] Figure 9a is an OCT image sample provided by an embodiment of the present application;
[0032] Figure 9b is a light attenuation coefficient image sample provided by an embodiment of the present application;
[0033] Fig.10 is a schematic diagram of an image processing device provided by an embodiment of the present application;
[0034] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0035] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0036] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0037] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0038] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0039] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0040] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0041] Optical Coherence Tomography (OCT) is an imaging technology. It uses the basic principle of weak coherent light interferometer to split the light emitted by the light source into two beams, one of which is emitted to the tissue to be tested, also called the sample arm, and the other is emitted to the reference reflector, also called the reference arm. Then, the two light signals reflected from the tissue to be tested and from the reference reflector are superimposed and interfered. Finally, the light signals are displayed in different grayscales according to the different tissues to be tested, thereby imaging the inside of the tissue.
[0042] The existing traditional optical coherence imaging technology has a weak ability to identify vulnerable plaques. Judging vulnerable plaques requires judging the thickness of the fibrous cap, which requires doctors to measure manually. Subjective factors of the measurer may lead to variability in the measurement results. Improving the recognition ability from the system and equipment side is difficult and will also incur high costs.
[0043] Due to the presence of dark areas such as lipids in vulnerable plaques, the fibrous cap boundary of the vulnerable plaque in the OCT image is not very clear. The display effect of existing OCT images is poor, which is not convenient for assisting medical staff to judge the load of vulnerable plaques.
[0044] The embodiment of the present application provides an image processing method, based on the result of AngioCoalesceRegistration (ACR), calculates the plaque attenuation index (IPA) value based on the light attenuation coefficient image, and maps the IPA value to the angiography image. Since the light attenuation in the area of vulnerable plaque is more obvious, the angiography image obtained by the image processing method provided in the embodiment of the present application can intuitively prompt the doctor that there may be vulnerable plaques in the current image frame by displaying the IPA value corresponding to the light attenuation image, making the display of the IPA value more intuitive and effective.
[0045] It should be noted that the image processing method provided in the embodiment of the present application can be implemented by software and / or hardware including but not limited to a device for acquiring angiography images, an OCT device, a local third-party computing device, and a remote third-party computing device, and the present application does not limit the subject implementing the image processing method. The third party is a device other than the OCT device and the device for acquiring angiography images.
[0046] Figure 1 The image processing method provided by the embodiment of the present application is shown. Figure 1 As shown, the method includes steps S110 to S150. The specific implementation principle of each step is as follows:
[0047] S110 , in the process of acquiring an optical coherence tomography (OCT) image group for the object to be investigated, acquiring an angiography image group for the object to be investigated.
[0048] In the process of performing optical coherence tomography on the blood vessels of the object of investigation, such as the coronary artery, by using an OCT device, a series of OCT images are obtained, which are recorded as an OCT image group. In this process, angiography imaging is performed on the object of investigation by an angiography imaging device to obtain a series of angiography images, which are recorded as angiography image group. In some embodiments, the angiography imaging is for the coronary artery, and the obtained image is called a coronary arteriography image (CAG).
[0049] It is understandable that the imaging device for acquiring the OCT image group and the angiography image group may be the same device, or may be two different devices, or may be a combination of devices in a control relationship. In some embodiments, the OCT image group and the angiography image group may be image processed by the OCT device, and the angiography imaging device may also perform image processing on the OCT image group and the angiography image group. In some embodiments, after the OCT image group and the angiography image group are acquired by the OCT device and the angiography imaging device, a third-party computing device acquires the OCT image group and the angiography image group through a storage medium, a communication cable, and a communication network for processing.
[0050] S120, registering the OCT image group and each angiography image in the angiography image group to obtain a registration parameter.
[0051] It should be understood that Figure 2 As shown, the OCT image group 21 is a series of tomographic images of the blood vessels of the object of investigation, which is equivalent to a cross-sectional image of the blood vessels. The angiography image 22 is a projection image of the blood vessels of the object of investigation. The OCT image group is registered with the angiography image, that is, the blood vessel cross section corresponding to each OCT image 211 in the OCT image group is determined, and the position 221 of the blood vessel projection in the angiography image is determined. In other words, a correspondence between each OCT image in the OCT image group and the position in the angiography image is established. The OCT image group and each angiography image in the angiography image group are registered, that is, for each angiography image in the angiography image group, a correspondence between the OCT image group and the position in the angiography image is established.
[0052] S130: Based on each OCT image in the OCT image group, generate a light attenuation coefficient image group corresponding to each OCT image.
[0053] The optical attenuation coefficient (OAC) of the vascular wall (including plaque) tissue is an optical characteristic parameter of OCT images. The optical attenuation coefficient of biological tissue varies with spatial position. Therefore, the optical attenuation coefficient can be used to quantitatively calibrate tissue components such as thin fibrous caps, calcifications, and lipid-rich plaques.
[0054] In some embodiments, the optical parameters of the OCT device, such as the Rayleigh length (zR), the half width of the roll-off function (zW), etc., can be used to calculate the light attenuation coefficient corresponding to the tissue in each OCT image based on the OCT image, and obtain a light attenuation coefficient image. The calculation method includes but is not limited to the curve fitting (CF) method or the depth-resolved (DR) model method. Thereby, a series of light attenuation coefficient images corresponding to the OCT image group are obtained, which are recorded as the light attenuation coefficient image group. It should be pointed out that there is a one-to-one correspondence between the OCT image group and the light attenuation coefficient image group. In other words, the registration parameters of the OCT image group and the angiography image group are consistent with the registration parameters of the light attenuation coefficient image group and the angiography image group.
[0055] S140, calculating the plaque attenuation index IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group.
[0056] In some embodiments, the index of plaque attenuation (IPA) is the fraction of pixels in the attenuation map whose attenuation coefficient is greater than a threshold value x. In a specific example, the fraction can also be multiplied by a coefficient of 1000. Specifically, the IPA value can be calculated using the following formula.
[0057]
[0058] Where x is the plaque attenuation coefficient threshold, μ t is the attenuation coefficient, N(μ t >x) is the total number of A-lines with a maximum attenuation value greater than x on each A-line, N total Indicates the number of all A lines.
[0059] S150 , based on the registration parameters, using the IPA value to mark each angiography image in the angiography image group to obtain a target angiography image group.
[0060] In some embodiments, the IPA value is a grayscale value, and each angiography image in the angiography image group is marked with the IPA value to obtain a target angiography image group, including: marking the grayscale value at a position corresponding to the light attenuation image on each angiography image.
[0061] In some embodiments, a correspondence between the IPA value and the marking parameter is preset, such as a comparison table or a conversion curve formula. The IPA value is mapped to the marking parameter using the correspondence. The marking parameter can be an RGB color value, or a computer code corresponding to a symbol such as "+", "*", "#", or a computer code corresponding to other marking symbols. The marking parameter corresponding to the IPA value is displayed at a position corresponding to the light attenuation image on each angiography image.
[0062] In some embodiments, the OCT image group is a cross-sectional image of the pullback vessel on the angiography image, that is, the OCT images in the OCT image group correspond to the position of the pullback vessel on the angiography image. It should be understood that due to the one-to-one correspondence between the optical attenuation coefficient image and the OCT image, the IPA value obtained from each optical attenuation coefficient image also corresponds to a position of the pullback vessel on the angiography image. Based on the registration parameters, each angiography image in the angiography image group is marked using the IPA value to obtain a target angiography image group, including: according to the correspondence between the preset IPA value and the marking parameter, each of the IPA values is converted into a marking parameter; according to the registration parameters, at the target position of each angiography image, the marking parameters of the corresponding optical attenuation coefficient image are displayed, and the target position is the pixel position on the pullback path corresponding to the optical attenuation coefficient image.
[0063] In a specific example, the marking parameter may be a color value. For example, if the IPA value is 100, the correspondence between the preset IPA value and the marking parameter is queried (which may be a comparison table), and it is determined that the RGB value corresponding to the IPA value is [179, 62, 110]. According to the registration parameter, the pixel at the position corresponding to the pullback blood vessel on the angiography image displays the RGB value. That is, the pullback path on the current angiography image is rendered and displayed in color.
[0064] It is understandable that if Figure 3 As shown, the embodiment of the present application obtains the registration parameters, that is, the correspondence, of the OCT image group and each angiography image by registering the angiography image group and each angiography image, so that the light attenuation coefficient image obtained by each OCT image has a consistent correspondence with each angiography image. On this basis, the IPA value obtained based on the light attenuation coefficient image is marked on each angiography image, so that the position and compliance of the vulnerable plaque can be clearly and intuitively observed, thereby improving the ability to identify vulnerable plaques of the explored object.
[0065] In the above Figure 1 The image processing method shown is based on Figure 4 As shown, step S120, registering the OCT image group and each angiography image in the angiography image group to obtain registration parameters, includes steps S121 to S123. This process is also called Angio Coalesce Registration (ACR).
[0066] S121, detecting the pullback path in each of the angiography images.
[0067] In angiography images, the pullback vessel is the vessel with the guidewire, that is, the vessel to be scanned. In some cases, the pullback is also called the pullback. The pullback path, also known as the pullback path, is the path scanned by the optical catheter during the OCT scanning process.
[0068] In some embodiments, Figure 5 As shown, detecting the pullback path in each angiography image includes steps S1211 to S1214:
[0069] S1211, using a pre-trained target detection model, detecting the pullback blood vessels in each of the angiography images.
[0070] Among them, the pulled-back blood vessel is the blood vessel with a guide wire, that is, the blood vessel to be scanned.
[0071] In some embodiments, the target detection model may be a deep learning network model for target detection. In some embodiments, the target detection model performs an image segmentation operation on the pullback vessels detected in the angiography image.
[0072] S1212, detecting the starting position and the ending position of the imaging target object in the pullback blood vessel in each of the angiography images.
[0073] The development target may be a development ring or an optical probe. The development ring is a metal ring provided at the end of the guide wire to increase the development effect.
[0074] S1213: Project the starting position and the ending position onto each of the angiography images using an iterative closest point (ICP) algorithm.
[0075] In some embodiments, in the process of acquiring an optical coherence tomography (OCT) image group for the object under investigation, an angiography image group for the object under investigation is acquired, the starting position is the position of the development ring movement of the first angiography image, and the ending position is the position of the development ring movement of the last angiography image. When the development ring positions of the first and last images are determined, the pull-back path of the development ring movement can be obtained, and then other angiography images can be projected through the ICP algorithm. In this way, each angiography image is marked with the starting position and the ending position.
[0076] S1214: Obtain a pullback path in each of the angiography images based on the starting position and the ending position in each of the angiography images using a shortest path algorithm.
[0077] In some embodiments, the weights matrix is established with the grayscale values of the pixels of the current angiography image. Since the grayscale value of the blood vessel portion is low, the weight is low, and the shortest path algorithm is to ensure that the path weight between the target points is the lowest. After determining the starting and ending positions of the development loop, the pullback path can be determined on the pullback vessel.
[0078] Figure 6 It is a schematic diagram of image registration provided by an embodiment of the present application. Figure 6 Taking the developing ring position of the last frame of the angiography image 61, that is, the end position 611 of the developing ring as an example, an example of projecting the developing ring position to other angiography images by the ICP algorithm is shown. And an example of obtaining the pull-back path by the shortest path method for any frame of angiography image 62 having the developing ring start position 612 and the developing ring end position 611 after the projection operation.
[0079] The registration parameters include the correspondence between each OCT image in the OCT image group and the target position, and the target position is the pixel position in each angiography image. It should be understood that the target position is the pixel position on the pullback path in each angiography image.
[0080] By registering the OCT image group and each angiography image in the angiography image group, the corresponding relationship between each OCT image in the OCT image group and the target position is obtained, so that the light attenuation coefficient image obtained using each OCT image has a consistent corresponding relationship with the target position. On this basis, the IPA value obtained based on the light attenuation coefficient image is marked at the target position of each angiography image, so that the position and compliance of the vulnerable plaque can be clearly and intuitively observed, thereby improving the ability to identify vulnerable plaques of the explored object.
[0081] S122, for each angiography image in the angiography image group, sampling the pullback path in the angiography image at equal intervals according to the frame rate of the OCT image group, to obtain a corresponding relationship between each OCT image and a pixel point position on the pullback path.
[0082] S123: Using the correspondence between each OCT image and the pixel position on the pull-back path as the registration parameter.
[0083] It should be noted that the default OCT pullback speed remains constant, so the distance moved per unit time is also constant. Therefore, the pullback path in the angiography image can be sampled at equal intervals to establish a corresponding relationship between the position on the pullback path and each OCT image.
[0084] In some embodiments, assuming that the pullback path on the angiography image is 600 pixels, a set of OCT pullbacks is 300 frames of images. The OCT device obtains one frame of OCT image each time it scans, and correspondingly, the position of the development ring on the angiography image moves by 2 pixels. Therefore, one OCT image corresponds to the position of two pixels on each angiography image.
[0085] Figure 6 Taking a frame of OCT image 64 as an example, the registration relationship between the OCT image and the angiography image 63 that determines the pullback path is that the frame number of the OCT image 64 corresponds to a position 631 of the returning blood vessel on the angiography image 63. It should be pointed out that the position 631 may contain one or more pixels, which depends on the OCT frame rate and the number of pixels contained in the pullback path.
[0086] exist Figure 5 Based on the image processing method shown, step S1211, using a pre-trained target detection model, detects the pullback vessels in each of the angiography images, including:
[0087] like Figure 7 As shown, a trained target detection model is used as the target detection model, namely, an Attention-U-net model.
[0088] The Attention-U-net model is trained using a sample set of angiography images with pre-labeled pullback vessels. In some embodiments, for the application of coronary angiography, a set of CAG angiography images can be prepared, in which experts label the pullback vessels with guidewires, for training the Attention-U-net model to identify the pullback vessels in the CAG images.
[0089] It should be pointed out that if the Attention-U-net network is used to perform fine segmentation of CAG images in various positions, mainly including the left anterior descending artery (LAD), left circumflex artery (LCX), and the right coronary artery (RCA), then when collecting training samples, the proportions of CAG images in various positions should be kept consistent as much as possible to ensure that the segmentation performance of the network in each model is close.
[0090] In some embodiments, since the data volume of the CAG image sample set of the pullback blood vessels annotated by experts is relatively small, the data sample volume can be expanded by rotating, flipping, adjusting contrast, etc.
[0091] Attention-U-net adds an attention mechanism to U-net, which implements the attention mechanism by supervising the features of the previous level through the features of the next level. Figure 7 As shown in Figure 2, the Attention-U-net includes the process of downsampling and upsampling, and adds an attention mechanism. Figure 7 Medium express.
[0092] The Attention-U-net model uses a loss function L that includes the weight parameter of the enlarged and pulled-back blood vessels. AUN train.
[0093]
[0094]
[0095] Among them, r ln represents the real pixel category of category l at the nth position. The categories in the embodiment of the present application are divided into two categories: pullback blood vessel path pixels and background pixels. ln represents the corresponding predicted probability value, ω l Represents the weight of each category. The larger the proportion of the category in the image, the smaller the weight.
[0096] It should be understood that, for the convenience of understanding the present application, the present application embodiment provides the above formula for an example of a loss function including a weight parameter for enlarging and pulling back the blood vessel. Those skilled in the art can refer to this example and adjust the form of the weight parameter for enlarging and pulling back the blood vessel and the specific parameters of the loss function in combination with actual conditions.
[0097] It can be understood that by automatically learning parameters to adjust the activation value, limiting the activated part to the segmented area, reducing the activation value of the background to optimize the segmentation, and achieving end-to-end segmentation, Attention-U-net has a higher segmentation accuracy in complex images, so that the trained model can automatically segment the blood vessels with guidewires passing through and quickly locate the retracted blood vessels.
[0098] In the above Figure 1 The image processing method shown is based on Figure 8 As shown, the embodiment of the present application also provides a method for generating a light attenuation coefficient image group corresponding to each OCT image in the OCT image group by using a cycle-consistent generative adversarial network (CycleGAN). The cycle-consistent generative adversarial network is also called a cycle-consistent generative adversarial network.
[0099] Refer to the method provided in the above embodiment for generating a light attenuation coefficient image group corresponding to each OCT image based on each OCT image in the OCT image group. Based on the optical parameters of the OCT device, the optical attenuation coefficient image of the OCT image is obtained by calculation, and the optical hardware parameters of the OCT device are required. However, when the OCT device leaves the factory, the relevant parameters may have some slight differences, which will cause the light attenuation coefficient image generated by the OCT image to have systematic errors. In order to eliminate the influence of the OCT system on the IPA calculation, the embodiment of the present application provides a method for synthesizing a light attenuation coefficient image based on the OCT image using a CycleGAN network, and calculating the IPA value based on the synthesized light attenuation coefficient image.
[0100] In some embodiments of the present application, OCT images generated by multiple OCT devices are prepared for the CycleGAN network to form an OCT image sample set. Based on the OCT image sample set, a light attenuation coefficient image sample set is obtained by calculating the optical parameters of each OCT device. Figure 9a What is shown is an OCT image sample provided by an embodiment of the present application. Figure 9b Shown is a light attenuation coefficient image sample provided by an embodiment of the present application.
[0101] Using OCT image sample sets and light attenuation coefficient image sample sets generated by multiple devices to train the CycleGAN network can improve the generalization ability of the CycleGAN network, so that the trained CycleGAN network can generate corresponding light attenuation coefficient images based on the OCT images generated by any OCT device.
[0102] In some embodiments, since the data volume of the OCT image sample set and the light attenuation coefficient image sample set obtained through expert annotation is small, the OCT image sample set and the light attenuation coefficient image sample set can be transformed by rotating, flipping, adjusting contrast, etc. to expand the data sample volume.
[0103] See also Figure 8 , the OCT image is synthesized into the light attenuation coefficient image using the CycleGAN network. The CycleGAN network mainly consists of two cycles, the forward cycle and the reverse cycle.
[0104] Among them, the forward loop mainly includes three independent CNN models, among which the synthesizer network is also called the generator network:
[0105] (1)Syn IPA The OCT image Img OCT Synthesizer network for IPA images;
[0106] (2)Syn OCT The light attenuation coefficient image Syn IPA (Img OCT ) A synthesizer network that converts back to OCT images;
[0107] (3)Dis IPA is to distinguish the synthetic light attenuation coefficient image Syn IPA (Img OCT ) and the real light attenuation coefficient image RealIPAImg discriminator network.
[0108] For the synthetic light attenuation coefficient image, the label is 0, and for the real light attenuation image, the label is 1. The discriminator network can distinguish between synthetic and real through continuous learning, that is, for synthetic, the discriminator output is 0, and for real, the output is 1. However, as the model is continuously trained, the quality of the generated light attenuation map is getting better and better, closer to the real one, and finally the discriminator can hardly distinguish between synthetic and normal, and the purpose of model training is achieved.
[0109] When the network is IPA Try to distinguish the synthetic light attenuation coefficient image Syn IPA (Img OCT ) and the real light attenuation coefficient image RealIPAImg, the network Syn IPA The OCT image will be synthesized as close to the real light attenuation coefficient image as possible. IPA (Img OCT ), making the network Dis IPA In addition, the synthetic light attenuation coefficient image Syn IPA (Img OCT) also needs to be synchronized through the network OCT Convert the OCT image back so that the original image Syn OCT (Img IPA ) to be reconstructed as accurately as possible.
[0110] In order to improve the stability of training, a reverse loop is added to synthesize OCT images using the optical attenuation coefficient image, and then convert the synthesized OCT image back to the optical attenuation coefficient image. The reverse loop also contains three parts, among which the two synthesis networks of the reverse loop are common to the forward loop, that is, the network Syn OCT and Network Syn IPA In addition, the reverse loop also includes the discriminator network Dis OCT , used to distinguish the synthetic OCT images Syn OCT (Img IPA ) and the real OCT image RealOCTImg.
[0111] The adversarial objectives of the synthesizer network and the discriminator network are reflected in the following loss function Loss IPA and Loss OCT middle.
[0112] Discriminator Dis IPA It is used to judge whether the image is a real light attenuation coefficient image. When the image is a real light attenuation coefficient image, the value is 1, and when the image is a synthetic light attenuation coefficient image, the value is 0. The discriminator Dis light attenuation coefficient will minimize the loss term as much as possible. IPA for:
[0113] Loss IPA =(1-Dis IPA (Img IPA )) 2 +Dis IPA (Syn IPA (Img OCT )) 2 .
[0114] Similarly, the discriminator Dis OCT Used to determine whether the image is a real OCT image, if it is real, it is 1, otherwise it is 0, and its loss is OCT for:
[0115] Loss OCT =(1-Dis OCT (Img OCT )) 2 +Dis OCT (Syn OCT (Img IPA )) 2.
[0116] In addition, when calculating the cycle consistency loss Loss cycle When , considering that both OCT images and light attenuation coefficient images focus on areas with larger grayscale values, that is, unstable plaques, but unstable plaques account for a very small proportion in the image, the weight coefficient of the area with larger grayscale values is increased in the loss term.
[0117]
[0118] Wherein, i represents the pixel value of the corresponding position of the real image and the synthesized image, and N represents the number of pixels in the image. In some embodiments, 704x704 is taken here. OCT (Syn IPA (Img OCT )) represents the conversion of the synthesized optical attenuation coefficient image back to the OCT image. IPA (Syn OCT (Img IPA )) represents the light attenuation coefficient image converted from the synthesized OCT image. When the pixel value at the corresponding position in the real image is larger, the instability it represents is greater, and the error at that position is smaller.
[0119] The cycle consistency loss of the cycleGAN network provided in the embodiment of the present application includes a first generation loss term and the second generation loss term The first generation loss term includes a first weight coefficient that is negatively correlated with the pixel value of the first generation sample The first generated sample is an OCT image sample; the second generated loss term includes a second weight coefficient that is negatively correlated with the pixel value of the second generated sample The second generated sample is a light attenuation coefficient image sample. Because the effective pixel area in OCT and IPA accounts for a small proportion of the image, if the weight term is not added, the weight of the loss on the effective area of OCT and IPA will be smaller, but through the weight coefficients in these two items, the smaller the proportion of effective pixels to the total pixels of the image, the larger these two items will be, thereby increasing the weight.
[0120] The total loss function of the cycleGAN network is Loss total Loss total =Loss IPA +Loss OCT +λLoss cycle . Among them, λ is the scaling factor, which is a hyperparameter.
[0121] Specific network structure diagram reference Figure 8, which includes a forward loop and a reverse loop. In the forward loop, Syn IPA Synthesize the OCT image into the light attenuation coefficient image. OCT The synthesized optical attenuation coefficient image is converted back to an OCT image close to the original image. IPA It is used to distinguish the real light attenuation coefficient image RealIPAImg from the synthesized light attenuation coefficient image. OCT is the optical attenuation coefficient image synthesized OCT image, Syn IPA Convert the synthesized OCT image back to an optical attenuation coefficient image close to the original image, Dis OCT It is used to distinguish the real OCT image RealOCTImg from the synthesized OCT image.
[0122] It should be pointed out that when training the CycleGAN network, the OCT image is trained to be synthesized into the light attenuation coefficient image, and the light attenuation coefficient image is synthesized into the OCT image, that is, both the forward cycle and the reverse cycle are trained. However, after CycleGAN is trained, only the OCT image is used to synthesize the light attenuation coefficient image part of the network.
[0123] It can be understood that the method provided in the embodiment of the present application, which uses the CycleGAN network to synthesize the light attenuation coefficient image according to the OCT image, and calculates the IPA value according to the synthesized light attenuation coefficient image, can get rid of the influence of specific OCT device parameters, directly synthesize the light attenuation coefficient image through the OCT image, eliminate the system error, and greatly improve the processing efficiency of the synthesized light attenuation coefficient image.
[0124] Corresponding to the above Figure 1 The image processing method shown, Fig.10 The figure shows an image processing device M100 provided in an embodiment of the present application, including:
[0125] The image acquisition module M110 is used to acquire a group of angiography images for the object to be investigated during the process of acquiring a group of optical coherence tomography (OCT) images for the object to be investigated.
[0126] The image registration module M120 is used to register the OCT image group and each angiography image in the angiography image group to obtain registration parameters.
[0127] The light attenuation coefficient image generating module M130 is used to generate a light attenuation coefficient image group of plaque attenuation coefficients corresponding to each OCT image in the OCT image group based on each OCT image in the OCT image group.
[0128] The IPA value generating module M140 is used to calculate the IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group.
[0129] The image marking module marks each angiography image in the angiography image group by using the IPA value based on the registration parameters to obtain a target angiography image group.
[0130] It can be understood that the various implementation modes and implementation combinations in the above embodiments and their beneficial effects are also applicable to this embodiment and will not be described in detail here.
[0131] Fig.11 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Fig.11 As shown, the electronic device D10 of this embodiment includes: at least one processor D100 ( Fig.11 Only one processor, a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100 are shown in the figure. When the processor D100 executes the computer program D102, the steps in any of the above-mentioned method embodiments are implemented.
[0132] The electronic device D10 may be an OCT device, angiography imaging device, a desktop computer, a notebook, a PDA, a cloud server or other computing device. The electronic device may include, but is not limited to, a processor D100 and a memory D101. Those skilled in the art will appreciate that Fig.11 This is only an example of the electronic device D10 and does not constitute a limitation on the electronic device D10. The electronic device D10 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.
[0133] The processor D100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0134] In some embodiments, the memory D101 may be an internal storage unit of the electronic device D10, such as a hard disk or memory of the electronic device D10. In other embodiments, the memory D101 may also be an external storage device of the electronic device D10, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device D10. Further, the memory D101 may also include both an internal storage unit of the electronic device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory D101 may also be used to temporarily store data that has been output or is to be output.
[0135] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0136] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0137] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0138] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0139] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0141] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0142] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0143] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0144] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0145] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for image processing, characterized in that: include: In the process of acquiring an optical coherence tomography (OCT) image group for the object under investigation, acquiring an angiography image group for the object under investigation; Registering the OCT image group and each angiography image in the angiography image group to obtain a registration parameter; Based on each OCT image in the OCT image group, generating a light attenuation coefficient image group corresponding to each OCT image; Calculating the plaque attenuation index IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group; Based on the registration parameters, each angiography image in the angiography image group is marked using the IPA value to obtain a target angiography image group.
2. The method according to claim 1, characterized in that Registering the OCT image group and each angiography image in the angiography image group to obtain registration parameters includes: detecting a pullback path in each of the angiographic images; For each angiography image in the angiography image group, sampling the pullback path in the angiography image at equal intervals according to the frame rate of the OCT image group, and obtaining a corresponding relationship between each OCT image and a pixel point position on the pullback path; The corresponding relationship between each OCT image and the pixel position on the pull-back path is used as the registration parameter.
3. The method according to claim 2, characterized in that Detecting the pullback path in each of the angiography images includes: Using a pre-trained target detection model, detecting the pullback blood vessels in each of the angiography images; In the pullback blood vessel in each of the angiographic images, detecting a starting position and an ending position of a development target object; Projecting the starting position and the ending position onto each of the angiography images using a closest point iteration algorithm; A pullback path in each of the angiography images is obtained by using a shortest path algorithm based on a starting position and an ending position in each of the angiography images.
4. The method according to claim 3, characterized in that The target detection model is an attention U-net model; The Attention-U-net model is trained using a sample set of angiography images with pre-labeled pullback vessels; The Attention-U-net model is trained using a loss function that includes a weight parameter for enlarging and pulling back blood vessels.
5. The method according to claim 1, characterized in that Based on each OCT image in the OCT image group, a plaque attenuation coefficient light attenuation coefficient image group corresponding to each OCT image is generated, including: A pre-trained recurrent generative adversarial network is used to obtain light attenuation coefficient images corresponding to each OCT image in the OCT image group based on the OCT images.
6. The method according to claim 5, characterized in that The recurrent generative adversarial network is trained using an OCT image sample set generated by multiple OCT devices and an optical attenuation coefficient image sample set determined according to the OCT image sample set; The loss function for training the cyclic generative adversarial network includes a cycle consistency loss; The cycle consistency loss includes a first generation loss term and a second generation loss term; The first generated loss term includes a first weight coefficient that is negatively correlated with a pixel value of a first generated sample; the first generated sample is the OCT image sample; The second generated loss term includes a second weight coefficient that is negatively correlated with a pixel value of a second generated sample; the second generated sample is the light attenuation coefficient image sample.
7. The method according to claim 2, characterized in that Based on the registration parameter, each angiography image in the angiography image group is marked by using the IPA value to obtain a target angiography image group, including: According to the preset correspondence between the IPA value and the marking parameter, each of the IPA values is converted into a marking parameter; According to the registration parameters, the marking parameters of the corresponding light attenuation coefficient image are displayed at the target position of each angiography image, and the target position is the pixel position on the pullback path corresponding to the light attenuation coefficient image.
8. An image processing device, characterized in that: include: An image acquisition module, used for acquiring a group of angiography images for the object under investigation during the process of acquiring a group of optical coherence tomography (OCT) images for the object under investigation; An image registration module, used for registering the OCT image group and each angiography image in the angiography image group to obtain registration parameters; An optical attenuation coefficient image generation module, used to generate a plaque attenuation coefficient optical attenuation coefficient image group corresponding to each OCT image based on each OCT image in the OCT image group; An IPA value generating module, used for calculating the IPA value of each of the light attenuation coefficient images in the light attenuation coefficient image group; The image marking module marks each angiography image in the angiography image group by using the IPA value based on the registration parameters to obtain a target angiography image group.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Vascular data processing and image registration systems, methods, and apparatuses
CN105188550A
Cardiovascular optical coherence tomography image enhancement method
CN105825488A
