Method for verifying multi-modal image registration and electronic device

Abstract

The application relates to the field of image processing, and provides a multi-modal image registration verification method, which comprises the following steps: acquiring an optical coherence tomography (OCT) image and a near-infrared autofluorescence (NIRAF) image corresponding to the OCT image; detecting the OCT image and the NIRAF image according to a target detection network to obtain first position information and second position information; determining an offset angle according to the first position information and the second position information; performing registration processing on the NIRAF image according to the offset angle to obtain a first NIRAF image; and performing verification processing on the first NIRAF image according to an optical attenuation coefficient image of the OCT image to obtain a verification result, which is used for indicating whether the first NIRAF image is registered with the OCT image. The verification method can verify the registration result of the image.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a multi-modal image registration verification method and an electronic device. BACKGROUND

[0002] In recent years, an Optical Coherence Tomography-Near-Infrared Autofluorescence (OCT-NIRAF) imaging system has appeared in the medical field. The OCT-NIRAF imaging system can perform multi-modal recognition on luminal tissue structure to obtain multi-modal images, i.e., to obtain Optical Coherence Tomography (OCT) images and corresponding Near-Infrared Autofluorescence (NIRAF) images.

[0003] Since there is a fixed included angle between the installation positions of a laser probe for collecting OCT images and a near-infrared laser probe for collecting NIRAF images in the OCT-NIRAF imaging system, when the OCT-NIRAF imaging system scans the luminal tissue structure, the OCT images and the NIRAF images cannot be obtained with pixel positions completely aligned. Before image fusion display, the OCT images and the corresponding NIRAF images need to be subjected to image registration processing. However, at present, after the OCT images and the NIRAF images are registered, image fusion is directly performed, and it is impossible to verify whether the images obtained after the image registration processing meet the registration requirements.

[0004] Therefore, how to verify the registration result of the images is a problem that needs to be solved urgently. SUMMARY

[0005] The present application provides a multi-modal image registration verification method and an electronic device, which can verify the registration result of the images.

[0006] In a first aspect, a method for verifying multi-modal image registration is provided, including: obtaining an optical coherence tomography (OCT) image and a near-infrared autofluorescence (NIRAF) image corresponding to the OCT image; detecting the OCT image and the NIRAF image based on a target detection network to obtain first position information and second position information, the first position information indicating a position of a guide wire in the OCT image, and the second position information indicating a position of the guide wire in the NIRAF image, the target detection network being configured to detect the position of the guide wire in the image; determining an offset angle based on the first position information and the second position information, the offset angle being a rotation angle between the position of the guide wire in the OCT image and the position of the guide wire in the NIRAF image; performing registration processing on the NIRAF image based on the offset angle to obtain a first NIRAF image; and performing verification processing on the first NIRAF image based on an optical attenuation coefficient image of the OCT image to obtain a verification result, the verification result indicating whether the first NIRAF image is registered with the OCT image.

[0007] The method can be performed by an electronic device or a chip in the electronic device. The electronic device detects positions of a guide wire in an OCT image and a NIRAF image based on a target detection network, and determines an offset angle of relative rotation between the OCT image and the NIRAF image based on first position information and second position information. Then, the electronic device performs image registration on the NIRAF image and the OCT image based on the offset angle to obtain a registered NIRAF image, i.e., a first NIRAF image. Then, the electronic device performs verification on the first NIRAF image based on an optical attenuation coefficient image of the OCT image to verify whether the first NIRAF image meets a registration requirement, thereby obtaining a verification result. Compared with the prior art in which image registration is performed based on only an offset angle, the present application provides a method for verifying multi-modal image registration, which can perform verification on a registered image after image registration processing, thereby determining whether the registered image meets a registration requirement.

[0008] Optionally, the verification processing on the first NIRAF image based on the optical attenuation coefficient image of the OCT image to obtain a verification result includes: calculating a difference between a maximum optical attenuation coefficient vector of the optical attenuation coefficient image and the first NIRAF image; in a case where the difference is greater than or equal to a first preset threshold, the verification result is that the first NIRAF image is not registered with the OCT image; and in a case where the difference is less than the first preset threshold, the verification result is that the first NIRAF image is registered with the OCT image.

[0009] In the embodiment, when the electronic device determines that the difference is greater than or equal to the first preset threshold, it indicates that the distance deviation of the pixel positions between the first NIRAF image and the OCT image is large, that is, the first NIRAF image and the OCT image are not registered; at this time, the detection result is that the first NIRAF image and the OCT image are not registered. When the electronic device determines that the difference is less than the first preset threshold, it indicates that the distance deviation of the pixel positions between the first NIRAF image and the OCT image is small, that is, the first NIRAF image and the OCT image are registered; at this time, the detection result is that the first NIRAF image and the OCT image are registered.

[0010] Optionally, the calculating the difference between the maximum light attenuation coefficient vector of the light attenuation coefficient image and the first NIRAF image comprises: calculating the maximum light attenuation coefficient vector according to the light attenuation coefficient image; generating a light attenuation intensity curve according to the maximum light attenuation coefficient vector; generating a NIRAF intensity curve according to the first NIRAF image; and determining the difference according to the light attenuation intensity curve and the NIRAF intensity curve.

[0011] In the embodiment, since the maximum light attenuation coefficient vector can accurately reflect the specific situation of the tissue structure (such as the atherosclerotic plaque in the blood vessel) in the lumen, the light attenuation intensity curve and the NIRAF intensity curve generated by the maximum light attenuation coefficient vector can be used to obtain the difference, so that the electronic device can verify whether the image after the registration of the OCT image and the NIRAF image meets the registration requirement according to the difference.

[0012] Optionally, the light attenuation intensity curve is an intensity curve generated according to the maximum light attenuation coefficient value in the clockwise rotation direction with the guide wire position in the OCT image as the starting position; or the light attenuation intensity curve is an intensity curve generated according to the maximum light attenuation coefficient value in the counterclockwise rotation direction with the guide wire position in the OCT image as the starting position.

[0013] Optionally, the NIRAF intensity curve is an intensity curve generated according to the pixel value in the clockwise rotation direction with the guide wire position in the first NIRAF image as the starting position; or the NIRAF intensity curve is an intensity curve generated according to the pixel value in the counterclockwise rotation direction with the guide wire position in the first NIRAF image as the starting position.

[0014] Optionally, according to the verification result and the first NIRAF image, a second NIRAF image is obtained, and the second NIRAF image is an image registered with the OCT image.

[0015] Optionally, the target detection network comprises a YOLO network or a Faster-RCNN network.

[0016] In a second aspect, an electronic device is provided, comprising a processor and a memory, the memory being configured to store a computer program, and the processor being configured to invoke and run the computer program from the memory, so that the electronic device performs the inspection method of any one of the first aspect.

[0017] In a third aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor performs the inspection method of any one of the first aspect.

[0018] The beneficial effects of the second and third aspects of the present application can be referred to the beneficial effects of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0020] Figure 1 The flowchart of the inspection method of multi-modal image registration in the embodiments of the present application;

[0021] Figure 2 The OCT image of the blood vessel in the embodiments of the present application;

[0022] Figure 3 The NIRAF image of the blood vessel in the embodiments of the present application;

[0023] Figure 4 The structural diagram of the electronic device in the embodiments of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, persons skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed description of well-known systems, devices and methods is omitted so as not to obscure the description of the present application with unnecessary details.

[0025] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0026] It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items, and that the term "at least one of' as used herein means "one, two, three, four, or more" of the listed items.

[0027] In addition, in the description of the present application and the appended claims, the terms "first", "second", "third", etc. are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0028] In the present application, the reference "one embodiment" or "some embodiments" and the like means that the specific features, structures or characteristics described in connection with the embodiment are included in one or more embodiments of the present application. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in further some embodiments" and the like appearing in different places in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. The terms "include", "contain", "have" and their variants mean "include but not limited to", unless otherwise specifically emphasized.

[0029] Due to the fixed included angle between the installation positions of the multi-modal probes in the Optical Coherence Tomography-Near-Infrared Autofluorescence (OCT-NIRAF) imaging system (i.e. the fixed included angle between the installation positions of the laser probe for collecting the OCT image and the near-infrared laser probe for collecting the NIRAF image), the OCT image and the NIRAF image output by the OCT-NIRAF imaging system after the intracavity scanning at the same time are not the scanning results of the same position in the intracavity. If the OCT image and the NIRAF image are fused and displayed without registration processing, the fusion image cannot accurately reflect the actual situation of the intracavity tissue structure (such as the atherosclerotic plaque in the blood vessel). Therefore, before the OCT image and the corresponding NIRAF image are fused and displayed, the OCT image and the corresponding NIRAF image need to be registered first. However, the prior art cannot determine whether the registration result of the OCT image and the corresponding NIRAF image (i.e. the multi-modal image) meets the registration requirement. Therefore, how to test the registration result of the image is a problem that needs to be solved urgently.

[0030] The present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0031] Figure 1is a flowchart of a method for verifying multi-modal image registration proposed in the present application, which can be executed by an electronic device or a chip on the electronic device. The method can be used to verify the registration result (i.e., the registered image) of the multi-modal images output by the OCT-NIRAF imaging system. The method comprises the following steps:

[0032] S101, obtaining an optical coherence tomography (OCT) image and a near-infrared autofluorescence (NIRAF) image corresponding to the OCT image.

[0033] Exemplarily, the electronic device obtains an OCT image and a NIRAF image corresponding to the OCT image, wherein the OCT image and the NIRAF image are multi-modal images obtained by the OCT-NIRAF imaging system after scanning different positions of the intraluminal tissue structure (such as the intravascular atherosclerotic plaque) at the same time.

[0034] S102, detecting the OCT image and the NIRAF image according to a target detection network to obtain first position information and second position information, the first position information being used to indicate the position of the guide wire in the OCT image, and the second position information being used to indicate the position of the guide wire in the NIRAF image, the target detection network being used to detect the position of the guide wire in the image.

[0035] Exemplarily, the intraluminal tissue is a blood vessel, Figure 2 the OCT image is a blood vessel, Figure 3 the NIRAF image is a blood vessel; Figure 2 In the above, 201 is a guide wire, and 202 is an atherosclerotic plaque in the blood vessel. Figure 3In the diagram, 301 represents the distribution of light intensity values ​​in the NIRAF image, and 302 represents atherosclerotic plaques within the blood vessel. The aforementioned target detection network is used to detect the position of the guidewire in both OCT and NIRAF images. When the laser probe acquiring OCT images (i.e., the OCT imaging catheter) or the near-infrared laser probe acquiring NIRAF images (i.e., the NIRAF imaging catheter) enters the body, the guidewire is placed into the blood vessel before the OCT or NIRAF imaging catheter. Then, the OCT or NIRAF imaging catheter hooks onto the guidewire and is placed into the blood vessel along the guidewire's path. Since the OCT or NIRAF imaging catheter itself is very soft, it is difficult to place it into the blood vessel without the guidewire; therefore, the guidewire can be considered as guiding the OCT or NIRAF imaging catheter. Because the metal guidewire can obstruct the imaging of blood vessels by the OCT or NIRAF imaging catheter, the guidewire and its shadow can be seen in the OCT or NIRAF images obtained after imaging the blood vessels through the OCT or NIRAF catheter. Furthermore, the pixel value of the area containing the guidewire in the OCT or NIRAF image is 0. Figure 2 and Figure 3 As shown, electronic devices can use a target detection network to identify the position of the guidewire in OCT images or NIRAF images, thereby achieving registration between OCT and NIRAF images.

[0036] The aforementioned object detection networks include either the YOLO network or the Faster-RCNN network. The YOLO network is a single neural network object detection system, built using GoogleNet as its architectural prototype with 24 convolutional layers and 2 fully connected layers. When detecting the location of guide wires in OCT or NIRAF images, the YOLO network needs to divide the input OCT or NIRAF image into blocks. For example, a 224*224 size OCT or NIRAF image can be divided into 7*7 cells, where each cell is 36*36 pixels. The YOLO network can mark a preset number of bounding boxes in cells containing guide wires. During guide wire recognition, the parameters that need to be determined include the bounding box (x, y, w, h), confidence score, and object probability (20 classes). (x, y) represents the center coordinates of the bounding box, and (w, h) represents the width and height of the bounding box. The YOLO network also needs to normalize the cell side lengths during guide wire recognition (i.e., normalize side lengths within the range of 0 to 1). The confidence score is defined as... If the cell contains an object (e.g., a guide wire), then Pr(Object) = 1; otherwise, Pr(Object) = 0. The indicates the proportion of the overlapping area of the predicted box and the real box to the total area of the predicted box and the real box. Since the YOLO network supports the detection of 20 types of objects, for each object (i.e., each object), the YOLO network outputs the probability of each object belonging to the 20 types of objects (i.e., object probability) respectively. The above-mentioned Faster-RCNN network needs to go through four processes of candidate region generation, feature extraction, classification and position refinement to identify the position of the guide wire in the OCT image or the NIRAF image during target detection. For example, first, the region of interest (ROI) is extracted according to target clustering, which is a region that may contain objects (such as guide wires); then the object type in each ROI region is identified.

[0037] As an optional embodiment, before the YOLO network identifies the position of the guide wire in the OCT image and the NIRAF image, the YOLO network needs to be trained first, and the training process of the YOLO network is as follows: (taking the OCT image containing atherosclerotic plaque in the blood vessel and the NIRAF image containing atherosclerotic plaque in the blood vessel as an example)

[0038] First, the electronic device obtains an OCT image sample containing atherosclerotic plaque (referred to as an OCT image sample) and a NIRAF image sample containing atherosclerotic plaque (referred to as a NIRAF image sample), wherein the OCT image sample and the NIRAF image sample both contain a guide wire. The above-mentioned OCT image sample is an image sample containing atherosclerotic plaque selected manually from OCT pullback data. The guide wire contained in the above-mentioned OCT image sample or NIRAF image sample is manually marked in advance. In the NIRAF image, different axial directions show different degrees of fluorescence decay, and each axial direction reflects the fluorescence decay of the blood vessel tissue through a normalized value between 0 and 1, and finally presents the decay spectrum of the cross-sectional plane of the blood vessel, and takes it as a parameter reflecting the distribution of atherosclerosis in the cross-sectional plane.

[0039] In order to obtain more training samples, data augmentation is needed to increase the number of OCT image samples and the number of NIRAF image samples. For example, the OCT image sample marked with the guide wire is augmented by rotating, adjusting the contrast and / or adding noise (such as random noise, salt and pepper noise, etc.) to increase the number of OCT image samples, and finally a sufficient number of OCT image samples containing the guide wire are obtained. The NIRAF image is augmented by rotating the NIRAF image to increase the number of NIRAF image samples. Since the NIRAF image has generated a parameter reflecting the transverse atherosclerosis, it is not necessary to adjust the contrast and add noise to the NIRAF image. Therefore, after the OCT image sample is rotated, the NIRAF image sample corresponding to the OCT image sample is also rotated, but when the OCT image sample is adjusted in contrast and noise is added, the NIRAF image sample does not need to be processed.

[0040] Secondly, the OCT image sample and the NIRAF image sample are respectively input into the initial YOLO network for iterative training.

[0041] During training, since the guide wire occupies a relatively small part of the pixels in the entire image, the weights of the positioning error and the classification error should not be equal, and in the selection of the loss function, a weighted loss function more suitable for small target recognition should be selected. Optionally, the loss weight of the boundary box coordinate prediction is increased, and the loss weight of the confidence prediction of the boundary box not containing the target is reduced. The loss function of the YOLO network is shown in the following formula:

[0042]

[0043] In the above formula, The OCT image sample and the NIRAF image sample are respectively input into the initial YOLO network for iterative training, and when the loss function meets the preset requirements, it indicates that the model has converged, that is, the initial YOLO network has completed training, and a trained YOLO network is obtained.

[0044] In different sizes of the predicted bounding box, in order to reduce the prediction deviation of the smaller predicted bounding box. The present application uses the square root of the width w and the height h of the predicted bounding box instead of the width w and the height h of the predicted bounding box to balance the offset of the smaller target detection prediction.

[0045] In the above formula, the electronic device calculates the Intersection over Union (IOU) error by using the above and the above ​The IOU error refers to the ratio of the intersection and the union of the predicted bounding box and the ground truth bounding box. The classification error is the output of the activation function. In addition to the coordinates, the electronic device calculates the loss function according to the error of the square root of the predicted bounding box width w and height h.

[0046] Finally, the two corresponding images (i.e., the OCT image and the NIRAF image) are input into the trained YOLO network, and the YOLO network accurately detects the guide wire positions in the OCT image and the NIRAF image, respectively, so that the electronic device can register the OCT image and the NIRAF image according to the offset angle of the guide wire positions in the OCT image and the NIRAF image.

[0047] For example, the electronic device uses the trained YOLO network to detect the positions of the guide wire in the OCT image and the NIRAF image, respectively. The YOLO network detects the position of the guide wire in the OCT image to obtain first position information, and the YOLO network detects the position of the guide wire in the NIRAF image to obtain second position information. The first position information and the second position information each include the coordinate position of the guide wire, which is used by the electronic device to determine the offset angle of the OCT image and the NIRAF image.

[0048] S103, determining the offset angle according to the first position information and the second position information, the offset angle being the rotation angle between the guide wire position in the OCT image and the guide wire position in the NIRAF image.

[0049] For example, the electronic device can use the first position information and the second position information output by the target detection network to calculate the offset angle between the position of the guide wire in the OCT image and the position of the guide wire in the NIRAF image. The electronic device can register the OCT image and the NIRAF image according to the offset angle. For example, the electronic device can use the position of the guide wire in the OCT image (i.e., the first position information) as a reference to rotate the NIRAF image by the offset angle to register the OCT image and the NIRAF image; or the electronic device can use the position of the guide wire in the NIRAF image (i.e., the second position information) as a reference to rotate the OCT image by the offset angle to register the OCT image and the NIRAF image.

[0050] S104, registering the NIRAF image according to the offset angle to obtain a first NIRAF image.

[0051] Exemplarily, the electronic device takes the position of the guidewire in the OCT image (i.e., the first position information) as a reference, and realizes the registration of the OCT image and the NIRAF image by rotating the NIRAF image by the above-mentioned offset angle. The electronic device rotates the NIRAF image by the above-mentioned offset angle to obtain a first NIRAF image; the first NIRAF image is an image obtained by the electronic device after preliminarily registering the NIRAF image with the OCT image as a reference.

[0052] In S105, the first NIRAF image is verified according to the light attenuation coefficient image of the OCT image to obtain a verification result, which is used to indicate whether the first NIRAF image is registered with the OCT image.

[0053] Exemplarily, since the OCT image is an image in polar coordinates, a light attenuation model in the polar coordinate system can be used to calculate the light attenuation coefficient of each pixel point of a single frame of the OCT image, and the light attenuation coefficient value corresponding to each pixel point is used to replace the value of each pixel point in the OCT image, so as to obtain a single frame of light attenuation coefficient image corresponding to a single frame of the OCT image in polar coordinates. The calculation formula of the above-mentioned light attenuation model is as follows:

[0054]

[0055]

[0056]

[0057] wherein, I0 is a scale factor, r represents the image depth, T(r) is a longitudinal point spread function, z0, z R , z c and z w respectively represent the beam waist position, the Rayleigh length, the scanning center point and the half-width of the roll-off function, and the values are 0, 3mm, 0 and 10um, respectively, u t is the light attenuation coefficient (i.e., the variable to be solved). Taking the logarithm of both sides of formula (1), and then using the least square method, the light attenuation coefficient u t can be calculated.

[0058] For example, the OCT machine scans the intraluminal tissue as a blood vessel. The OCT machine collects a single-frame OCT image of the blood vessel as 642x500, where 500 is the total number of A-lines scanned by the guide tube in the blood vessel during 360° scanning, and 642 is the number of pixel points scanned on each A-line. The light attenuation coefficient of each pixel point in the single-frame OCT image is calculated using the above light attenuation model to obtain a single-frame light attenuation coefficient image corresponding to the single-frame OCT image. The single-frame light attenuation coefficient image is also 642x500, where 500 is the total number of A-lines scanned by the guide tube in the blood vessel during 360° scanning, and 642 is the number of light attenuation coefficient values u on each A-line. t .

[0059] After the electronic device calculates the light attenuation coefficient image of the OCT image according to the above light attenuation coefficient model, it checks whether the first NIRAF image meets the registration requirement according to the light attenuation coefficient image of the OCT image. The above registration requirement includes that the distance deviation of the pixel position between the first NIRAF image and the OCT image is less than a preset distance. When the electronic device checks that the first NIRAF image meets the registration requirement according to the light attenuation coefficient image of the OCT image, it means that the distance deviation of the pixel position between the first NIRAF image and the OCT image is small (i.e., the distance deviation of the pixel position between the first NIRAF image and the OCT image is less than the preset distance); that is, the first NIRAF image is registered with the OCT image; at this time, the detection result is that the first NIRAF image is registered with the OCT image.

[0060] When the electronic device checks that the first NIRAF image does not meet the registration requirement according to the light attenuation coefficient image of the OCT image, it means that the distance deviation of the pixel position between the first NIRAF image and the OCT image is large (i.e., the distance deviation of the pixel position between the first NIRAF image and the OCT image is greater than or equal to the preset distance); that is, the first NIRAF image is not registered with the OCT image; at this time, the detection result is that the first NIRAF image is not registered with the OCT image.

[0061] Exemplarily, the first NIRAF image is checked according to the light attenuation coefficient image of the OCT image to obtain a checking result, including: calculating the difference between the maximum light attenuation coefficient vector of the light attenuation coefficient image and the first NIRAF image; in the case that the difference is greater than or equal to a first preset threshold, the checking result is that the first NIRAF image is not registered with the OCT image; in the case that the difference is less than the first preset threshold, the checking result is that the first NIRAF image is registered with the OCT image.

[0062] Since the light attenuation coefficient image of the single-frame OCT image has 500 A-lines and each A-line has 642 light attenuation coefficient values, the electronic device calculates the maximum light attenuation coefficient value of each A-line of the light attenuation coefficient image, and the 500 A-lines have 500 maximum light attenuation coefficient values, which form a 1x500 maximum light attenuation coefficient vector, that is, the single-frame light attenuation coefficient image can obtain a 1x500 maximum light attenuation coefficient vector.

[0063] Since the single-frame NIRAF image is also 642x500, where 500 is the total number of A-lines scanned by the guide tube in the 360° scanning of the intraluminal tissue (such as blood vessels), and 642 is the number of pixel points scanned on each A-line, and the pixel values corresponding to the 642 pixel points on each A-line are the same, the electronic device can select any pixel value from the 642 pixel points on each A-line to represent the light intensity value on the A-line. Therefore, for the single-frame NIRAF image, there are 500 A-lines, and finally 500 light intensity values can be obtained. For the first NIRAF image, there are also 500 A-lines, and finally 500 light intensity values can be obtained.

[0064] The electronic device calculates the light attenuation coefficient image of the OCT image using the light attenuation coefficient model, and then calculates the maximum light attenuation coefficient vector of the light attenuation coefficient image. The electronic device calculates the difference between each maximum light attenuation coefficient value in the maximum light attenuation coefficient vector (i.e., the 500 maximum light attenuation coefficient values) and each light intensity value in the 500 light intensity values in the first NIRAF image, and the total number of differences is 500. If the difference calculated by the electronic device is negative, the electronic device determines the absolute value of the negative number as the difference. The first preset threshold refers to the maximum light intensity difference allowed between each maximum light attenuation coefficient value in the 500 maximum light attenuation coefficient values and each light intensity value in the 500 light intensity values. When the electronic device determines that a preset number (such as 300 or 400) of the 500 differences are greater than or equal to the first preset threshold, or when the electronic device determines that the average of the 500 differences is greater than or equal to the first preset threshold, the electronic device determines that the first NIRAF image and the OCT image do not meet the registration requirement (i.e., the distance deviation between the pixel positions of the first NIRAF image and the OCT image is greater than or equal to the preset distance), that is, the test result is that the first NIRAF image and the OCT image are not registered.

[0065] In this embodiment, when the electronic device determines that the difference is greater than or equal to the first preset threshold, it indicates that the distance deviation of the pixel positions between the first NIRAF image and the OCT image is large; that is, the first NIRAF image and the OCT image are not registered; at this time, the detection result is that the first NIRAF image and the OCT image are not registered.

[0066] Exemplarily, in the case where the difference is less than the first preset threshold, the test result is that the first NIRAF image and the OCT image are registered.

[0067] When the electronic device determines that a preset number (such as 450 or 400) of the above-mentioned 500 differences are all less than the first preset threshold, or when the electronic device determines that the average value of the above-mentioned 500 differences is less than the first preset threshold, the electronic device determines that the first NIRAF image and the OCT image meet the registration requirement (that is, the distance deviation of the pixel positions between the first NIRAF image and the OCT image is less than the preset distance), that is, the test result is that the first NIRAF image and the OCT image are registered.

[0068] In this embodiment, when the electronic device determines that the difference is less than the first preset threshold, it indicates that the distance deviation of the pixel positions between the first NIRAF image and the OCT image is small; that is, the first NIRAF image and the OCT image are registered, at this time, the detection result is that the first NIRAF image and the OCT image are registered.

[0069] Exemplarily, the difference between the maximum light attenuation coefficient vector of the light attenuation coefficient image and the first NIRAF image is calculated, including: calculating the maximum light attenuation coefficient vector according to the light attenuation coefficient image; generating a light attenuation intensity curve according to the maximum light attenuation coefficient vector; generating a NIRAF intensity curve according to the first NIRAF image; determining the difference according to the light attenuation intensity curve and the NIRAF intensity curve.

[0070] The electronic device calculates a maximum light attenuation coefficient vector according to the light attenuation coefficient image, the maximum light attenuation coefficient vector having a total of 500 maximum light attenuation coefficient values; and the electronic device generates a light attenuation intensity curve according to the 500 maximum light attenuation coefficient values, wherein the light attenuation intensity curve is an intensity curve generated according to the maximum light attenuation coefficient values in the clockwise rotation direction with the guidewire position in the OCT image (or the light attenuation coefficient image) as the starting position; or the light attenuation intensity curve is an intensity curve generated according to the maximum light attenuation coefficient values in the counterclockwise rotation direction with the guidewire position in the OCT image (or the light attenuation coefficient image) as the starting position. Since the pixel value of the position where the guidewire is located in the OCT image or the light attenuation coefficient image is 0, and the 500 maximum light attenuation coefficient values are distributed in a circular ring shape on the image, the light attenuation intensity curve can be an intensity curve generated according to the maximum light attenuation coefficient values in the clockwise rotation direction or the counterclockwise rotation direction with the position where the guidewire is located as the starting position.

[0071] As known from the foregoing, the first NIRAF image has a total of 500 A lines, and finally 500 light intensity values can be obtained; the electronic device generates a NIRAF intensity curve according to the 500 light intensity values, wherein the NIRAF intensity curve is an intensity curve generated according to the pixel values in the clockwise rotation direction with the guidewire position in the first NIRAF image as the starting position; or the NIRAF intensity curve is an intensity curve generated according to the pixel values in the counterclockwise rotation direction with the guidewire position in the first NIRAF image as the starting position. Since the pixel value of the position where the guidewire is located in the first NIRAF image is 0, and the 500 light intensity values are distributed in a circular ring shape on the image, the NIRAF intensity curve can be an intensity curve generated according to the light intensity values in the clockwise rotation direction or the counterclockwise rotation direction with the position where the guidewire is located in the first NIRAF image as the starting position. In this embodiment, since the maximum light attenuation coefficient vector can accurately reflect the specific situation of the tissue structure (such as the atherosclerotic plaque in the blood vessel) in the lumen, the light attenuation intensity curve and the NIRAF intensity curve generated by the maximum light attenuation coefficient vector can be subtracted to obtain a difference value, so that the electronic device can verify whether the image after registration of the OCT image and the NIRAF image meets the registration requirement according to the difference value.

[0072] Exemplarily, according to the verification result and the first NIRAF image, a second NIRAF image is obtained, the second NIRAF image being an image registered with the OCT image.

[0073] For example, when the detection result indicates that the first NIRAF image is registered with the OCT image, it means that the distance deviation between the pixel positions of the first NIRAF image and the OCT image is less than the preset distance (i.e., the first NIRAF image is a NIRAF image meeting the registration requirement); at this time, the second NIRAF image can be the same as the first NIRAF image.

[0074] For example, when the detection result indicates that the first NIRAF image is not registered with the OCT image, it means that the distance deviation between the pixel positions of the first NIRAF image and the OCT image is greater than or equal to the preset distance (i.e., the first NIRAF image is a NIRAF image not meeting the registration requirement); at this time, the second NIRAF image can be different from the first NIRAF image, and the electronic device can perform re-registration processing or fine tuning on the first NIRAF image to obtain the second NIRAF image.

[0075] For example, the electronic device can perform re-registration processing or fine tuning on the first NIRAF image to obtain the second NIRAF image, which can include but is not limited to:

[0076] The electronic device can re-acquire the NIRAF image and repeat the above registration process, or the electronic device can re-adjust the registration result of the first NIRAF image and the OCT image to obtain the second NIRAF image meeting the registration requirement; or the electronic device can determine a new offset angle based on the first position information and the second position information, perform registration processing on the NIRAF image based on the new offset angle, and obtain the first NIRAF image, etc.

[0077] It should be understood that the above is for illustration only, and the present application does not make any limitation on the specific implementation manner of obtaining the second NIRAF image based on the detection result and the first NIRAF image.

[0078] In the embodiments of the present application, the first NIRAF image can be processed based on the detection result to obtain a second NIRAF image meeting the registration requirement; in one possible implementation manner, the first NIRAF image meets the registration requirement, and the first NIRAF image is an image after the registration with the OCT image is completed, and at this time, the second NIRAF image is the same as the first NIRAF image; in another possible implementation manner, the first NIRAF image does not meet the registration requirement, and the first NIRAF image can be adjusted so that the adjusted image, i.e., the second NIRAF image, meets the image registration requirement, thereby ensuring the accuracy of the image registration processing.

[0079] In summary, the electronic device detects the positions of the guidewire in the OCT image and the NIRAF image respectively through the target detection network, and determines the offset angle of the relative rotation of the OCT image and the NIRAF image according to the first position information and the second position information; then, the electronic device performs image registration on the NIRAF image and the OCT image according to the offset angle, to obtain the registered NIRAF image, i.e., the first NIRAF image; and then performs inspection on the first NIRAF image according to the light attenuation coefficient image of the OCT image, to inspect whether the first NIRAF image meets the registration requirement, so as to obtain an inspection result. Compared with the prior art in which image registration is performed only according to the offset angle, the present application provides a multi-modal image registration inspection method, which can perform inspection on the registered image after image registration processing, so as to determine whether the registered image meets the registration requirement.

[0080] It should be understood that, in the embodiments of the present application, the OCT image is taken as a reference to perform image registration processing on the NIRAF image; the image registration method of the present application is also applicable to taking the NIRAF image as a reference to perform image registration processing on the OCT image, and the present application does not make any limitation in this regard.

[0081] Figure 4 A structure schematic diagram of an electronic device provided by the present application is shown. Figure 4 The dashed line in the figure indicates that the unit or the module is optional. The electronic device 400 can be used to implement the methods described in the above method embodiments. The electronic device 400 can be a server or a chip.

[0082] The electronic device 400 includes one or more processors 401, which can support the electronic device 400 to implement the methods in the corresponding method embodiments. The processor 401 can be a general-purpose processor or a special-purpose processor. For example, the processor 401 can be a central processing unit (CPU). The CPU can be used to control the electronic device 400, execute software programs, and process data of the software programs. The electronic device 400 can also include a communication unit 405 to implement input (reception) and output (transmission) of signals. Figure 1

[0083] For example, the electronic device 400 can be a chip, the communication unit 405 can be an input and / or output circuit of the chip, or the communication unit 405 can be a communication interface of the chip, and the chip can be used as a component of the electronic device.

[0084] For another example, the communication unit 405 can be a transceiver of the electronic device 400, or the communication unit 405 can be a transceiver circuit of the electronic device 400.

[0085] ​The electronic device 400 can include one or more memories 402, on which programs 404 are stored, the programs 404 can be run by the processor 401 to generate instructions 403, so that the processor 401 executes the method described in the above method embodiments according to the instructions 403. Optionally, the memory 402 can also store data. Optionally, the processor 401 can also read the data stored in the memory 402, and the data can be stored in the same storage address as the program 404, or the data can be stored in a different storage address from the program 404.

[0086] The processor 401 and the memory 402 can be separately arranged, or can be integrated together, for example, integrated on a system on chip (SOC) of the electronic device.

[0087] The specific manner in which the processor 401 executes the inspection method of the multi-modal image registration can be referred to the related description in the method embodiments.

[0088] It should be understood that each step of the above method embodiments can be completed by a logic circuit in the form of hardware or an instruction in the form of software in the processor 401. The processor 401 can be a CPU, a digital signal processor (DSP), a field programmable gate array (FPGA), or other programmable logic devices, for example, discrete gates, transistor logic devices, or discrete hardware components.

[0089] The present application also provides a computer program product, which is executed by the processor 401 to implement the method of any method embodiment in the present application.

[0090] The computer program product can be stored in the memory 402, for example, the program 404, which is finally converted into an executable object file that can be executed by the processor 401 through preprocessing, compiling, assembling, and linking processes.

[0091] The present application also provides a computer readable storage medium, on which a computer program is stored, and the computer program is executed by a computer to implement the method of any method embodiment in the present application. The computer program can be a high-level language program or an executable object program.

[0092] The computer readable storage medium is, for example, the memory 402. The memory 402 can be a volatile memory or a nonvolatile memory, or the memory 402 can include both volatile and nonvolatile memory. The nonvolatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM), which is used as the external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DRRAM).

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and apparatuses and the generated technical effects can refer to the corresponding processes and technical effects in the foregoing method embodiments, and will not be described here.

[0094] In several embodiments provided in the present application, the disclosed system, device and method can be implemented in other ways. For example, some features of the above-described method embodiments can be omitted or not executed. The above-described device embodiments are merely illustrative, and the division of units is merely a logical function division. In actual implementation, another division manner can be used, and multiple units or components can be combined or integrated into another system. In addition, the coupling between units or the coupling between components can be direct coupling or indirect coupling, and the above-mentioned coupling includes electrical, mechanical or other forms of connection.

[0095] The above examples are only used to illustrate the technical solutions of the present application, but not to limit it. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for verifying multimodal image registration, characterized in that, The testing method includes: Acquire optical coherence tomography (OCT) images and near-infrared autofluorescence imaging (NIRAF) images, wherein the NIRAF images are images corresponding to the OCT images; The target detection network is used to detect the OCT image and the NIRAF image to obtain first position information and second position information. The first position information is used to indicate the position of the guide wire in the OCT image, and the second position information is used to indicate the position of the guide wire in the NIRAF image. The target detection network is used to detect the position of the guide wire in the image. The offset angle is determined based on the first position information and the second position information. The offset angle is the rotation angle between the guidewire position in the OCT image and the guidewire position in the NIRAF image. The NIRAF image is registered according to the offset angle to obtain a first NIRAF image; Calculate the maximum optical attenuation coefficient vector based on the optical attenuation coefficient image; Generate an optical attenuation intensity curve based on the maximum optical attenuation coefficient vector; Generate an NIRAF intensity curve based on the first NIRAF image; The difference is determined based on the light decay intensity curve and the NIRAF intensity curve; If the difference is greater than or equal to a first preset threshold, the test result is that the first NIRAF image and the OCT image are not registered. If the difference is less than a first preset threshold, the test result is that the first NIRAF image and the OCT image are registered.

2. The testing method according to claim 1, characterized in that, The light attenuation intensity curve is an intensity curve generated based on the maximum light attenuation coefficient value in the clockwise rotation direction, starting from the guide wire position in the OCT image; or, the light attenuation intensity curve is an intensity curve generated based on the maximum light attenuation coefficient value in the counterclockwise rotation direction, starting from the guide wire position in the OCT image.

3. The testing method according to claim 1, characterized in that, The NIRAF intensity curve is an intensity curve generated based on the pixel values ​​in a clockwise rotation direction, starting from the guidewire position in the first NIRAF image; or, the NIRAF intensity curve is an intensity curve generated based on the pixel values ​​in a counterclockwise rotation direction, starting from the guidewire position in the first NIRAF image.

4. The test method according to any one of claims 1 to 3, characterized in that, Also includes: Based on the test results and the first NIRAF image, a second NIRAF image is obtained, wherein the second NIRAF image is an image registered with the OCT image.

5. The test method according to any one of claims 1 to 3, characterized in that, The target detection network includes either the YOLO network or the Faster-RCNN network.

6. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store computer programs, and the processor being used to retrieve and run the computer programs from the memory, causing the electronic device to perform the testing method according to any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, A computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the testing method according to any one of claims 1 to 5.

Images