Anterior segment OCT image three-dimensional reconstruction method and device, computer equipment and medium

Abstract

The invention relates to the technical field of image processing, and provides an anterior segment OCT image three-dimensional reconstruction method and device, computer equipment and a medium, and the method comprises the steps: carrying out the semantic segmentation of a collected anterior segment full-circumference multi-angle OCT image sequence, and automatically recognizing a key region; and the subsequent processing can be more accurately carried out for the areas. Image registration is carried out on a semantic segmentation image sequence obtained through segmentation, an anterior segment full-circumference semantic segmentation image is obtained, and the aim of registration is to align images shot at different angles to form a coherent whole. Next, a target area contour is extracted from the anterior segment full-circumference semantic segmentation image, finally, the target area contour is projected to a three-dimensional space to form a contour point cloud of each structure of the anterior segment, and the point cloud can accurately represent the surface morphology of each structure of the anterior segment; therefore, high-precision three-dimensional reconstruction of the surface of each structure of the anterior segment is realized based on the contour point cloud of each structure of the anterior segment.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, computer equipment and medium for three-dimensional reconstruction of anterior segment OCT images. Background Art

[0002] As a non-invasive, high-resolution imaging method, optical coherence tomography (OCT) of the anterior segment has gradually become an important tool for anterior segment structural analysis. Compared with traditional ophthalmic ultrasound biomicroscopy (UBM), anterior segment OCT uses the principle of light interference to provide clearer imaging and more accurate quantitative analysis of key structures such as the cornea, iris, ciliary body and anterior chamber.

[0003] Anterior segment OCT has a variety of scanning modes, including line scanning, surface scanning, radial scanning and raster scanning. Among them, the surface scanning mode can quickly obtain high-quality tomographic images of the anterior segment of the eye, providing two-dimensional information for the diagnosis and treatment of anterior segment diseases. However, the anterior segment image generated in the surface scanning mode is a tomographic image, which has poor intuitiveness and limited information, and is limited to showing the structural details of a single section. Therefore, it is not conducive to comprehensive analysis. Relying solely on a single or a few tomographic images for diagnosis is not only difficult to form a comprehensive understanding of the condition of the anterior segment of the eye, but may also increase the risk of misdiagnosis or missed diagnosis due to incomplete information. Summary of the invention

[0004] Based on this, it is necessary to propose a method, device, computer equipment and medium for three-dimensional reconstruction of anterior segment OCT images in order to address the technical problems of poor intuitiveness and limited information in the prior art of anterior segment OCT images. This application aims to achieve three-dimensional reconstruction of the surfaces of key structures such as the cornea, iris and lens based on full-circle multi-angle OCT images of the anterior segment, providing doctors with more three-dimensional, complete and comprehensive anterior segment information. When assisting doctors in examining the condition of the anterior segment and diagnosing related ophthalmic diseases, they can make more in-depth and detailed disease analysis and treatment decisions based on more accurate and comprehensive three-dimensional views.

[0005] The first aspect of the present application provides a method for three-dimensional reconstruction of anterior segment OCT images, the method comprising:

[0006] Performing image semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence;

[0007] Performing image registration on the semantic segmentation image sequence to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye;

[0008] Extracting the target area contour from the anterior segment full circumference semantic segmentation image;

[0009] Projecting the target area contour into three-dimensional space to obtain a point cloud of contours of various structures in the anterior segment of the eye;

[0010] Based on the point cloud of the contours of the structures in the anterior segment of the eye, three-dimensional reconstruction is performed to obtain an anterior segment of the eye OCT three-dimensional image.

[0011] Optionally, performing image semantic segmentation on the anterior segment OCT image sequence to obtain a semantically segmented image sequence includes:

[0012] Perform image semantic segmentation on the anterior segment OCT image sequence through an OCT image semantic segmentation model based on U-Net or U2-Net to obtain a semantic segmentation image sequence;

[0013] The downsampling operation of the U-Net or the U2-Net uses a convolution operation with a step size of 2.

[0014] Optionally, performing image registration on the semantic segmentation image sequence to obtain an anterior segment full-circle semantic segmentation image includes:

[0015] Selecting an anterior segment OCT image from the anterior segment OCT image sequence as a reference OCT image, and selecting images other than the reference OCT image from the anterior segment OCT image sequence as OCT images to be registered;

[0016] Calculate a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered;

[0017] Performing pixel transformation on each semantic segmentation image in the semantic segmentation image sequence based on the rigid body transformation matrix to obtain a registered semantic segmentation image sequence;

[0018] The semantic segmentation image sequence after the registration is merged to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye.

[0019] Optionally, the calculating a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered includes:

[0020] Initialize the transformation parameters of the rigid body transformation matrix;

[0021] In each iteration, a rigid body transformation is performed on the OCT image to be registered according to the current transformation parameters, and the similarity between the transformed image and the reference OCT image is calculated;

[0022] Using a preset registration algorithm, updating transformation parameters to minimize the similarity;

[0023] When the convergence conditions are met, the final calculated rigid body transformation matrix is ​​output.

[0024] Optionally, projecting the target area contour into three-dimensional space to obtain a point cloud of contours of structures of the anterior segment of the eye includes:

[0025] Get the angle value of each semantic segmentation image;

[0026] Acquire a plurality of contour points and pixel coordinates of each contour point from the target area contour;

[0027] According to the angle value and the pixel coordinates, the target area contour is projected into a cylindrical coordinate system to obtain a point cloud of contours of various structures of the anterior segment of the eye.

[0028] Optionally, projecting the target area contour into a cylindrical coordinate system according to the angle value and the pixel coordinates to obtain a point cloud of contours of structures of the anterior segment of the eye includes:

[0029] Determine a height value in the cylindrical coordinate system according to a first specified component coordinate in the pixel coordinate;

[0030] Determine the radius value in the cylindrical coordinate system according to the second specified component coordinate in the pixel coordinate;

[0031] Obtain three-dimensional coordinates according to the radius value, the angle value and the height value;

[0032] Based on the three-dimensional coordinates, a point cloud is obtained.

[0033] Optionally, the method further includes:

[0034] Cache the anterior segment OCT image and the corresponding image identifier in a memory;

[0035] An image reconstruction request is sent to Redis according to the image identifier, and an anterior segment OCT image corresponding to the image identifier and stored in the memory and returned by Redis is obtained.

[0036] A second aspect of the present application provides a device for three-dimensional reconstruction of anterior segment OCT images, the device comprising:

[0037] A semantic segmentation module is used to perform semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence;

[0038] An image registration module is used to perform image registration on the semantic segmentation image sequence to obtain an anterior segment full circumference semantic segmentation image;

[0039] A contour extraction module, used to extract the target area contour from the anterior segment full circumference semantic segmentation image;

[0040] A point cloud acquisition module is used to project the target area contour into three-dimensional space to obtain a point cloud of the contours of each structure of the anterior segment of the eye;

[0041] The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the contour point cloud of each structure of the anterior segment of the eye to obtain an anterior segment of the eye OCT three-dimensional image.

[0042] The third aspect of the present application provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for three-dimensional reconstruction of anterior segment OCT images when executing the computer program.

[0043] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for three-dimensional reconstruction of anterior segment OCT images are implemented.

[0044] This application first performs semantic segmentation on the collected full-circle multi-angle OCT image sequence of the anterior segment of the eye, and can automatically identify and distinguish key areas such as the cornea, iris, and lens in the image, so that subsequent processing can be more accurately performed on these areas. Then, the semantic segmentation image sequence obtained by segmentation is image registration to obtain a full-circle semantic segmentation image of the anterior segment of the eye. The purpose of registration is to align images taken at different angles to form a coherent whole. Next, the contour of the target area is extracted from the full-circle semantic segmentation image of the anterior segment of the eye, which clearly reflects the shape and position of each structure of the anterior segment of the eye. Finally, the contour of the target area is projected into three-dimensional space to form a point cloud of the contours of each structure of the anterior segment of the eye. The point cloud is a collection of discrete points in three-dimensional space, which can accurately represent the surface morphology of each structure of the anterior segment of the eye, thereby realizing high-precision three-dimensional reconstruction of the surface of each structure of the anterior segment of the eye based on the point cloud of the contours of each structure of the anterior segment of the eye. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 It is a flow chart of the method for three-dimensional reconstruction of anterior segment OCT images provided in an embodiment of the present application.

[0047] Figure 2 It is a schematic diagram of the semantic segmentation of OCT images provided in an embodiment of the present application.

[0048] Figure 3 It is a structural diagram of the U-Net model provided in the embodiment of the present application.

[0049] Figure 4 It is a structural diagram of the U2-Net model provided in an embodiment of the present application.

[0050] Figure 5 It is a schematic diagram of solving the rigid transformation matrix in OCT image registration provided in an embodiment of the present application.

[0051] Figure 6 It is a schematic diagram of the registration of the semantic segmentation image provided in the embodiment of the present application.

[0052] Figure 7 It is a schematic diagram of extracting the contours of various structures of the anterior segment of the eye provided in an embodiment of the present application.

[0053] Figure 8 It is a point cloud diagram of the contours of various structures of the anterior segment of the eye provided in an embodiment of the present application.

[0054] Fig. 9 This is the three-dimensional reconstruction result of the surface of each structure of the anterior segment of the eye provided by the embodiment of the present application.

[0055] Fig.10 It is a schematic diagram of the interaction between the anterior segment surface three-dimensional reconstruction algorithm and the OCT device provided in the embodiment of the present application.

[0056] Fig.11 It is a functional module diagram of the anterior segment surface three-dimensional reconstruction device provided in an embodiment of the present application.

[0057] Fig.12 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0059] Figure 1 This is a flow chart of a method for three-dimensional reconstruction of anterior segment OCT images provided in an embodiment of the present application. The method for three-dimensional reconstruction of anterior segment OCT images includes the following steps.

[0060] S11, performing image semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence.

[0061] In order to comprehensively evaluate the anterior segment structure, it is necessary to perform multi-angle or continuous scanning of the anterior segment through an OCT (Optical Coherence Tomography) device to generate a series of OCT images. This series of OCT images (i.e., anterior segment OCT image sequence) is arranged in the scanning order of the OCT device to form the three-dimensional structural information of the anterior segment. The anterior segment includes structures such as the cornea, sclera, iris, and lens.

[0062] In order to obtain comprehensive information about the anterior segment structure and ensure that all important anterior segment structures are covered, this application obtains multi-angle OCT images of the entire anterior segment to avoid information loss or distortion caused by single-angle shooting.

[0063] The embodiment of the present application uses a deep learning semantic segmentation model to perform image semantic segmentation on the anterior segment OCT image. The deep learning semantic segmentation model is pre-trained based on the anterior segment OCT image dataset. After the deep learning semantic segmentation model is trained, one anterior segment OCT image can be input into the deep learning semantic segmentation model one at a time for image semantic segmentation, or all anterior segment OCT images can be input into the deep learning semantic segmentation model in batches for image semantic segmentation. Through the deep learning semantic segmentation model, different structures in the anterior segment OCT image (such as the cornea, iris, lens, etc.) can be automatically identified and labeled, thereby outputting the semantic segmentation result of the anterior segment OCT image, that is, an image with each structure labeled. The image output by the deep learning semantic segmentation model is called a semantically segmented image. Figure 2 As shown in the figure, the structure of the upper area in each semantic segmentation image is the corneal structure, the structure of the middle area is the iris structure, and the structure of the lower area is the lens structure. The semantic segmentation images output by the deep learning semantic segmentation model are arranged in the scanning order of the OCT device to form a semantic segmentation image sequence.

[0064] In some optional implementations, U-Net and U2-Net can be selected as the network framework of the deep learning semantic segmentation model, and a multi-classification cross entropy loss function (Cross-Entropy Loss Function) can be selected as the loss function of the deep learning semantic segmentation model.

[0065] See also Figure 3 and Figure 4 As shown, Figure 3 It is the structure diagram of U-Net. Figure 4The following is the structure diagram of U2-Net. The U-Net network adopts the U-shaped structure of encoder-decoder. U2-Net has made improvements on the basis of U-Net, replacing the downsampling layer in the original U-Net encoder and the upsampling layer in the decoder with small residual U-blocks (RSU). The difference from the ordinary U-Net structure is that in addition to retaining the downsampling encoder, upsampling decoder and jump connection, RSU also sets the initial input convolution layer at the initial stage of the module to extract local features.

[0066] The encoder is used to extract the features of the anterior segment OCT image. It consists of convolution operations and downsampling operations, and includes a 5-layer structure. In each layer of the structure, the encoder first performs two convolution operations on the input feature map of the layer to extract richer features. The purpose of the downsampling operation is to generate a thumbnail of the image. The maximum pooling operation is usually used, which is simple to calculate and can better preserve the image features. The decoder is used to repair the details and spatial dimensions of the feature map. It consists of convolution operations, upsampling operations, and skip-connection operations, and also includes a 5-layer structure. Similar to the encoder, in each layer of the structure, the decoder first performs two convolution operations on the input feature map of the layer. The decoder will then upsample the result of the convolution operation to restore the size of the image. The upsampling methods include transposed convolution, nearest neighbor interpolation, bilinear interpolation, etc.

[0067] In traditional U-Net or U2-Net, the downsampling process uses a pooling operation (such as maximum pooling or average pooling), which can effectively reduce the dimension of the feature map, but this process is often not learnable and may cause a certain degree of information loss. Therefore, in order to reduce information loss, the downsampling operation of the embodiment of the present application uses a convolution operation with a stride of 2, that is, a convolution operation with a stride of 2 is used instead of a pooling operation.

[0068] In the above optional implementation, downsampling is achieved through a convolution operation with a step size of 2. Since the parameters of the convolution operation can be adjusted through learning, the network can automatically adjust the mode and degree of downsampling according to different tasks and data sets, thereby more flexibly retaining feature information useful for subsequent tasks. In addition, the convolution operation can not only reduce information loss, but also capture more details and contextual information compared to the pooling operation, thereby improving the accuracy of the segmentation task.

[0069] In an optional embodiment, in order to further improve the accuracy of the segmentation task, the upsampling operation in the embodiment of the present application uses nearest neighbor interpolation. Nearest neighbor interpolation fills each pixel in the target image by finding the pixel value of the nearest neighbor, which has a small amount of calculation and a fast speed. In addition, in the edge area, the edge information of the original image can be better maintained, which helps to maintain the edge clarity of the segmentation result.

[0070] The automatic, efficient and precise segmentation of key structures such as the cornea, iris, and lens in the anterior segment OCT images can lay a solid foundation for subsequent three-dimensional reconstruction, ensuring that the final reconstructed anterior segment structure surface is both accurate and realistic.

[0071] S12, performing image registration on the semantic segmentation image sequence to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye.

[0072] In practical applications, since the anterior segment OCT images come from different time points, there may be obvious geometric differences between these images, such as position offset, angle rotation, etc. The image obtained by semantic segmentation only classifies a single anterior segment OCT image, but does not solve the problem of geometric differences between different anterior segment OCT images. Therefore, in order to fuse multi-source information and obtain a more comprehensive and accurate anterior segment full-circle semantic segmentation image, after semantic segmentation of each anterior segment OCT image in the anterior segment OCT image sequence, it is necessary to perform registration of the semantic segmentation image sequence.

[0073] Semantic segmentation image sequence registration refers to the process of spatially aligning a series of multi-angle OCT images taken around the anterior segment of the eye (these images together cover the entire circumference of the anterior segment of the eye) so that these images can be compared and analyzed in the same coordinate system. Image registration can eliminate the geometric differences between different anterior segment OCT images caused by factors such as shooting angle, instrument artifacts, or involuntary axial movement of the eye during shooting, so that different anterior segment OCT images can be fused together. The registered anterior segment OCT image sequence can more accurately reflect the three-dimensional structure of the anterior segment of the eye, providing a reliable basis for subsequent three-dimensional reconstruction.

[0074] Optionally, performing image registration on the semantic segmentation image sequence to obtain an anterior segment full-circle semantic segmentation image includes:

[0075] Selecting an anterior segment OCT image from the anterior segment OCT image sequence as a reference OCT image, and selecting images other than the reference OCT image from the anterior segment OCT image sequence as OCT images to be registered;

[0076] Calculate a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered;

[0077] Performing pixel transformation on each semantic segmentation image in the semantic segmentation image sequence based on the rigid body transformation matrix to obtain a registered semantic segmentation image sequence;

[0078] The semantic segmentation image sequence after the registration is merged to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye.

[0079] Since the anterior segment OCT image sequence covers the original information of multiple structures of the anterior segment, an anterior segment OCT image can be selected from the anterior segment OCT image sequence as a reference OCT image. The reference OCT image acts as a benchmark or "anchor point" in image registration. The goal of registration is to transform other anterior segment OCT images in the anterior segment OCT image sequence into the same space or coordinate system as the reference OCT image so that they can be compared or fused with each other.

[0080] The anterior segment OCT image with the highest image quality (such as the best clarity, contrast, noise level, etc.) can be selected as the reference OCT image to ensure the accuracy of the registration. Alternatively, an image located at the center or a specified position in the anterior segment OCT image sequence can be selected as the reference OCT image.

[0081] For the registration of multi-angle OCT images of the entire anterior segment, the registration algorithm is required to ensure that the OCT images to be registered will not be drastically deformed after registration, that is, the operations performed by the registration algorithm on the OCT images to be registered are preferably translation and rotation, that is, rigid body transformation. Rigid body transformation is a special affine transformation that keeps the Euclidean distance between two points in the image unchanged, that is, it only includes translation, rotation and reflection (in two-dimensional cases) or translation and rotation (in three-dimensional cases, without reflection).

[0082] In an optional implementation, the calculating and obtaining a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered includes:

[0083] Initialize the transformation parameters of the rigid body transformation matrix;

[0084] In each iteration, a rigid body transformation is performed on the OCT image to be registered according to the current transformation parameters, and the similarity between the transformed image and the reference OCT image is calculated;

[0085] Using a preset registration algorithm, updating transformation parameters to minimize the similarity;

[0086] When the convergence conditions are met, the final calculated rigid body transformation matrix is ​​output.

[0087] The preset registration algorithm may be an improved automatic sub-pixel registration (Marquardt-Levenberg) algorithm, which can constrain affine transformation to a rigid body transformation, thereby preventing the registered image from being deformed.

[0088] For each OCT image to be registered, initialize the parameters of its rigid transformation matrix and the parameters of the improved Marquardt-Levenberg algorithm. The parameters of the rigid transformation matrix include: the initial values ​​of the rotation angle (θ) and the translation vector (Δu = [Δx, Δy]). The initial value can be random or set based on some prior knowledge. The parameters of the improved Marquardt-Levenberg algorithm include: the number of iterations, convergence conditions (such as mean square error threshold), learning rate, etc.

[0089] See also Figure 5 As shown, the reference OCT image and each OCT image to be registered are input into the improved Marquardt-Levenberg algorithm, and the algorithm will find their respective rigid transformation matrices, which define how to transform the corresponding OCT image to be registered into the space of the reference OCT image so as to minimize the similarity between the two. Similarity is a measure of the degree of similarity between two images in content or structure, and determines how to adjust (or "register") the OCT image to be registered so that it is aligned with the reference OCT image. The improved Marquardt-Levenberg algorithm uses the pixel intensity value as the feature of the image, and uses the mean square intensity difference between the reference OCT image and the OCT image to be registered in the transformed coordinates as an objective criterion for measuring the similarity between the two. The smaller the mean square difference, the more similar the two images are in intensity, and the larger the mean square difference, the less similar the two images are in intensity. It should be understood that the mean square difference is used as an inverse indicator of similarity, that is, images with high similarity correspond to lower mean square difference values.

[0090] In each iteration, the improved Marquardt-Levenberg algorithm performs a rigid transformation on the OCT image to be registered according to the current transformation parameters (rotation angle and translation vector), and calculates the intensity mean square error between the transformed image and the reference OCT image to adjust or update the transformation parameters by minimizing the intensity mean square error. The above process is repeated until the convergence condition is met (such as the intensity mean square error is less than the preset threshold, or the maximum number of iterations is reached). When the algorithm converges, the final calculated rigid transformation matrix is ​​output. The rigid transformation matrix contains the rotation angle and translation vector required to transform the OCT image to be registered to align with the reference OCT image. The rigid transformation matrix finally calculated is the optimal rigid transformation matrix.

[0091] After finding the optimal rigid transformation matrix, the rigid transformation matrix is ​​applied to transform (or "register") the entire semantic segmentation image sequence, such as Figure 6 As shown, each semantic segmentation image in the semantic segmentation image sequence can be aligned with the reference OCT image. Since the semantic segmentation image is directly generated from the original anterior segment OCT image and has the same spatial correspondence with the anterior segment OCT image, the calculated rigid body transformation matrix can be used to perform pixel-level transformation on each semantic segmentation image in the semantic segmentation image sequence, ensuring that the pixels in the semantic segmentation image can be aligned with the corresponding areas in the reference OCT image, thereby eliminating the deviation caused by different shooting angles or positions.

[0092] Finally, all semantically segmented images after pixel transformation and registration are merged into a complete semantically segmented image of the entire periphery of the anterior segment. During the merging process, the boundaries between adjacent images can be smoothly transitioned through interpolation, fusion algorithms, or manual adjustments to avoid obvious stitching marks.

[0093] The above implementation method can ensure that the original features and details of the anterior segment OCT image are retained as much as possible during the registration process by solving the rigid body matrix based on the original anterior segment OCT image sequence, thereby improving the accuracy of the registration.

[0094] S13, extracting the target area contour from the anterior segment full-circle semantic segmentation image.

[0095] Each pixel in the anterior segment full-circle semantic segmentation image is assigned a category label, indicating which specific anatomical structure or tissue type the pixel belongs to. In the anterior segment full-circle semantic segmentation image, different colors or grayscale values ​​are usually used to distinguish different anatomical structures.

[0096] Extracting the target region contour from the anterior segment full circumference semantic segmentation image is to identify and outline the boundary of the region of interest (ROI). The target region in the embodiment of the present application refers to a specific anatomical part of the anterior segment structure, such as the cornea, iris, lens, etc.

[0097] After determining the target area, edge detection algorithms, such as the Canny edge detection algorithm and the Sobel edge detection algorithm, can be used to detect points with drastic brightness changes in the full-circle semantic segmentation image of the anterior segment of the eye to identify edges and obtain the contours of the structures of the anterior segment of the eye. Figure 7 As shown, it is a schematic diagram of the contours of various structures extracted from the full-circle semantic segmentation image of the anterior segment of the eye, wherein the moon-shaped area in the upper part is the corneal contour, the rectangular area in the lower part is the lens contour, and the area between the corneal contour and the lens contour is two iris contours.

[0098] In some embodiments, morphological operations such as erosion, dilation, opening and closing operations can be used to post-process the extracted contours to eliminate noise, fill holes, smooth edges, optimize shapes, or segment connected areas to more accurately extract contours.

[0099] S14, projecting the target area contour into three-dimensional space to obtain a point cloud of contours of various structures of the anterior segment of the eye.

[0100] In order to obtain a complete representation of the anterior segment structure in three-dimensional space, it is necessary to project the target area contour into three-dimensional space to obtain the contour point cloud of each structure of the anterior segment, such as Figure 8 shown.

[0101] A point cloud is a collection of points in three-dimensional space, each of which represents a specific spatial location. In the three-dimensional reconstruction of the anterior segment of the eye, these points usually correspond to specific locations on the surface of the structure.

[0102] In an optional embodiment, projecting the target area contour into three-dimensional space to obtain a point cloud of contours of structures of the anterior segment of the eye includes:

[0103] Get the angle value of each semantic segmentation image;

[0104] Acquire a plurality of contour points and pixel coordinates of each contour point from the target area contour;

[0105] According to the angle value and the pixel coordinates, the target area contour is projected into a cylindrical coordinate system to obtain a point cloud of contours of various structures of the anterior segment of the eye.

[0106] To generate the point cloud, each point on the 2D contour is converted into a 3D coordinate according to its position in the image and the corresponding depth information.

[0107] The angle value usually refers to the rotation angle relative to a fixed reference system (such as the patient's head or eyes) when each image is taken. The OCT device has the function of recording the shooting angle during scanning, and the angle value associated with each anterior segment OCT image can be obtained from the output file or metadata of the OCT device, thereby obtaining the angle value of the semantic segmentation image corresponding to each anterior segment OCT image.

[0108] Extract multiple key points (contour points) from the contour of the target area (such as iris, cornea, lens, etc.) and record the pixel coordinates of each contour point. Multiple contour points can be obtained by uniform sampling, that is, a certain number of points are uniformly distributed along the contour line as contour points. You can also use image processing algorithms (such as corner point detection) to extract feature points on the contour as contour points.

[0109] Since the structure of the anterior segment of the eye is roughly cylindrical, the target area contour can be projected into a cylindrical coordinate system according to the angle value and the pixel coordinates of the obtained contour points to obtain the contour point cloud of each structure of the anterior segment of the eye. The center point of the image or a fixed reference point is usually selected as the origin of the cylindrical coordinate system.

[0110] In an optional implementation, projecting the target area contour into a cylindrical coordinate system according to the angle value and the pixel coordinates to obtain a point cloud of contours of structures of the anterior segment of the eye includes:

[0111] Determine a height value in the cylindrical coordinate system according to a first specified component coordinate in the pixel coordinate;

[0112] Determine the radius value in the cylindrical coordinate system according to the second specified component coordinate in the pixel coordinate;

[0113] Obtain three-dimensional coordinates according to the radius value, the angle value and the height value;

[0114] Based on the three-dimensional coordinates, a point cloud is obtained.

[0115] Calculate the radius value: According to the pixel coordinates of the contour point (especially the x coordinate, assuming that the image is placed horizontally), calculate the horizontal distance from each point to the origin. This distance is the radius value r in the cylindrical coordinate system.

[0116] The angle value of each semantic segmentation image is taken as the θ value in the cylindrical coordinate system.

[0117] The first specified component coordinate may be a y coordinate, and correspondingly, the second specified component coordinate is an x ​​coordinate. Since the image is two-dimensional, but the height value in the cylindrical coordinate system is depth information. The embodiment of the present application directly determines the y coordinate (i.e., height H) in the pixel coordinates of the contour point as the height value z of each contour point in the cylindrical coordinate system.

[0118] The x coordinate (i.e., width W) in the pixel coordinates of the contour point is converted to the radius value in the cylindrical coordinate system. The absolute value obtained by subtracting half of the width of the semantic segmentation image from the width W of the contour point is used as the radius value r in the cylindrical coordinate system, i.e., r = |WW I / 2|, where W I is the width of the semantic segmentation image. The purpose is to convert the x component in the two-dimensional coordinates into a radius value with the center of the image as the origin, so as to adapt to the representation of the cylindrical coordinate system.

[0119] The radius value r, height value z, and angle value θ corresponding to each contour point are combined to form the three-dimensional coordinates [r, θ, z] in the cylindrical coordinate system. The [r, θ, z] coordinates of all contour points are combined to form the contour point cloud of each structure of the anterior segment of the eye.

[0120] S15, performing three-dimensional reconstruction based on the contour point cloud of each structure of the anterior segment of the eye to obtain an anterior segment of the eye OCT three-dimensional image.

[0121] Medical image 3D reconstruction refers to the process of constructing a 3D model of a tissue or organ using a series of 2D tomographic images. This application uses the contour information of the anterior segment structures (iris, cornea, lens, etc.) provided by the registered anterior segment full-circle semantic segmentation image to reconstruct the complete surface of the anterior segment structures, such as Fig. 9 As shown, a more intuitive virtual anatomical entity is provided to doctors to facilitate correct diagnosis and treatment of relevant diseases.

[0122] In an optional implementation, in order to improve the accuracy and quality of three-dimensional reconstruction, the contour point cloud of each structure of the anterior segment of the eye may be preprocessed, including noise removal, hole filling, smoothing, etc.

[0123] A 3D surface can be constructed by connecting the points in the point cloud into a triangular mesh to generate a smooth and continuous surface model. A smooth 3D surface can also be generated by local fitting and interpolation of the point cloud.

[0124] After the 3D surface reconstruction is completed, the mesh can be optimized to remove redundant triangles, improve the mesh topology and increase rendering efficiency. The optimized 3D mesh is rendered and visualized using 3D rendering software to generate high-quality anterior segment OCT 3D images, such as Fig. 9 The anterior segment OCT three-dimensional image can be viewed and analyzed from different angles and perspectives, providing doctors with more intuitive and comprehensive anterior segment structure information.

[0125] See also Fig.10 , which is a logic diagram of interaction between an OCT device host and an OCT device provided in an embodiment of the present application.

[0126] In order to achieve rapid interaction between the anterior segment OCT image 3D reconstruction algorithm and the OCT device, this application uses the Django framework. Django is an advanced Python web framework that allows rapid development of secure and maintainable websites. In this scenario, the Django framework is used to encapsulate the anterior segment OCT image 3D reconstruction algorithm. That is, the anterior segment OCT image 3D reconstruction algorithm is packaged into a service that can be accessed through a Web interface. In this way, the OCT device can interact with the anterior segment OCT image 3D reconstruction algorithm through network requests without directly processing complex algorithm logic.

[0127] Specifically, the anterior segment OCT image 3D reconstruction algorithm is encapsulated in a system developed using the Django framework. This system can be an independent server or server cluster. The server can be a physical server or a virtual machine running in a cloud environment, running a Django application and providing a Web interface for interacting with the anterior segment OCT image 3D reconstruction algorithm.

[0128] The OCT device scans the anterior segment OCT image, and the OCT device host (client) obtains the anterior segment OCT image from the OCT device and sends the anterior segment OCT image to the server via a HyperText Transfer Protocol (HTTP) request. The server (Django application) receives the image from the OCT device host through Django's views, processes the received image using a 3D reconstruction algorithm written in Python, generates a 3D point cloud or mesh model, and stores it in a database or directly saves it in the form of a file.

[0129] The server returns the 3D reconstruction result (which may be a point cloud file, mesh file or rendered image) to the OCT device host through HTTP response. The OCT device host receives the 3D data returned by the server and uses an appropriate library (such as Three.js) to display the 3D image on a web page.

[0130] Generally speaking, the anterior segment OCT image scanned by the OCT device is first saved to the hard disk, and then read from the hard disk to the memory for processing. However, saving to the hard disk will increase the delay of data processing because the hard disk has a low read and write speed. Therefore, in order to speed up the efficiency of image access, in an optional embodiment, before S11, the method further includes:

[0131] Cache the anterior segment OCT image and the corresponding image identifier in a memory;

[0132] An image reconstruction request is sent to Redis according to the image identifier, and an anterior segment OCT image corresponding to the image identifier and stored in the memory and returned by Redis is obtained.

[0133] Every time the OCT device scans a new anterior segment OCT image, the OCT device host generates a unique image identifier (Key) for the anterior segment OCT image and caches it together with the anterior segment OCT image in the Redis memory database. Redis is a high-performance key-value storage system that supports multiple types of data structures, and all data is stored in memory, so the access speed is very fast. Key can be any string used to identify and access data in the Redis cache.

[0134] When the anterior segment OCT image needs to be processed, an image reconstruction request is sent directly to Redis through the corresponding Key, and Redis will quickly return the anterior segment OCT image corresponding to the Key stored in the memory.

[0135] The above optional implementation method reduces hard disk read and write operations and data processing delays by using Redis to cache the anterior segment OCT images and allows the anterior segment OCT image three-dimensional reconstruction algorithm to call the anterior segment OCT images directly from the memory, thereby reducing the time required for the reconstruction process, improving the real-time nature and processing efficiency of the reconstruction, and enabling doctors to obtain the three-dimensional reconstruction results of the anterior segment of the eye more quickly, thereby making more accurate diagnosis and treatment.

[0136] The following describes the beneficial effects of the anterior segment OCT image 3D reconstruction method provided by the embodiment of the present application:

[0137] This application first performs semantic segmentation on the collected full-circle multi-angle OCT image sequence of the anterior segment of the eye, and can automatically identify and distinguish key areas such as the cornea, iris, and lens in the image, so that subsequent processing can be more accurately performed on these areas. Then, the semantic segmentation image sequence obtained by segmentation is image registration to obtain a full-circle semantic segmentation image of the anterior segment of the eye. The purpose of registration is to align images taken at different angles to form a coherent whole. Next, the contour of the target area is extracted from the full-circle semantic segmentation image of the anterior segment of the eye, which clearly reflects the shape and position of each structure of the anterior segment of the eye. Finally, the contour of the target area is projected into three-dimensional space to form a point cloud of the contours of each structure of the anterior segment of the eye. The point cloud is a collection of discrete points in three-dimensional space, which can accurately represent the surface morphology of each structure of the anterior segment of the eye, thereby realizing high-precision three-dimensional reconstruction of the surface of each structure of the anterior segment of the eye based on the point cloud of the contours of each structure of the anterior segment of the eye.

[0138] The registration of this application is achieved by solving the rigid body transformation matrix. Stable image registration can ensure the spatial consistency of images taken at different angles, effectively avoid three-dimensional reconstruction errors caused by image misalignment, and can cope with OCT images collected by equipment in different environments, ensuring the stability and consistency of three-dimensional reconstruction results with strong robustness.

[0139] This application can accurately represent the surface morphology of various structures in the anterior segment of the eye through contour extraction and point cloud generation, ensuring the accuracy of the three-dimensional reconstruction results.

[0140] Fig.11 It is a functional module diagram of the anterior segment OCT image three-dimensional reconstruction device provided in an embodiment of the present application.

[0141] In some embodiments, the anterior segment OCT image 3D reconstruction device 110 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the anterior segment OCT image 3D reconstruction device 110 may be stored in a memory of a computer device and executed by at least one processor to perform (see Figure 1 Description) Function of three-dimensional reconstruction of anterior segment OCT images.

[0142] In this embodiment, the anterior segment OCT image three-dimensional reconstruction device 110 can be divided into multiple functional modules according to the functions it performs. The functional modules may include: a semantic segmentation module 1101, an image registration module 1102, a contour extraction module 1103, a point cloud acquisition module 1104, a three-dimensional reconstruction module 1105 and an image cache module 1106. The module referred to in this application refers to a series of computer-readable instruction segments that can be executed by at least one processor and can perform fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0143] The semantic segmentation module 1101 is used to perform image semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence;

[0144] The image registration module 1102 is used to perform image registration on the semantic segmentation image sequence to obtain an anterior segment full circumference semantic segmentation image;

[0145] The contour extraction module 1103 is used to extract the target area contour from the anterior segment full circumference semantic segmentation image;

[0146] The point cloud acquisition module 1104 is used to project the target area contour into three-dimensional space to obtain the contour point cloud of each structure of the anterior segment of the eye;

[0147] The three-dimensional reconstruction module 1105 is used to perform three-dimensional reconstruction based on the contour point cloud of each structure of the anterior segment of the eye to obtain an anterior segment of the eye OCT three-dimensional image.

[0148] The image caching module 1106 is used to cache the anterior segment OCT image and the corresponding image identifier in the memory; send an image reconstruction request to Redis according to the image identifier, and obtain the anterior segment OCT image returned by Redis and stored in the memory and corresponding to the image identifier.

[0149] It should be understood that the various variations and specific embodiments of the anterior segment OCT image three-dimensional reconstruction method provided in the above-mentioned embodiments are also applicable to the anterior segment OCT image three-dimensional reconstruction device in the present embodiment. Through the detailed description of the above-mentioned anterior segment OCT image three-dimensional reconstruction method, those skilled in the art can clearly know the implementation process of the anterior segment OCT image three-dimensional reconstruction device in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0150] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, all or part of the steps of the method for three-dimensional reconstruction of anterior segment OCT images are implemented.

[0151] See also Fig.12 , which is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. In a preferred embodiment of the present application, the computer device 12 includes a memory 1201 , at least one processor 1202 , and at least one communication bus 1203 .

[0152] Those skilled in the art should understand that Fig.12 The structure of the computer device shown does not constitute a limitation of the embodiments of the present application. The computer device 12 may also include more or less other hardware or software than shown in the figure, or a different arrangement of components.

[0153] In some embodiments, the computer device 12 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The computer device 12 may also include client devices, which include but are not limited to any electronic product that can interact with a client through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0154] It should be noted that the computer device 12 is only an example, and other existing or future electronic products that are suitable for the present application should also be included in the protection scope of the present application and included here by reference.

[0155] In some embodiments, the memory 1201 stores a computer program, and when the computer program is executed by the at least one processor 1202, all or part of the steps in the three-dimensional reconstruction method of the anterior segment OCT image are implemented. The memory 1201 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data. Further, the computer-readable storage medium may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function, and the like.

[0156] In some embodiments, the at least one processor 1202 is the control core (Control Unit) of the computer device 12, and uses various interfaces and lines to connect various components of the entire computer device 12, and executes various functions and processes data of the computer device 12 by running or executing programs or modules stored in the memory 1201, and calling data stored in the memory 1201. For example, when the at least one processor 1202 executes the computer program stored in the memory, it implements all or part of the steps of the three-dimensional reconstruction method of the anterior segment OCT image described in the embodiment of the present application; or implements all or part of the functions of the three-dimensional reconstruction device of the anterior segment OCT image. The at least one processor 1202 can be composed of an integrated circuit, for example, it can be composed of a single packaged integrated circuit, or it can be composed of multiple integrated circuits with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips.

[0157] In some embodiments, the at least one communication bus 1203 is configured to realize connection and communication between the memory 1201 and the at least one processor 1202, etc. Although not shown, the computer device 12 may also include a power supply (such as a battery) for powering each component. Preferably, the power supply may be logically connected to the at least one processor 1202 through a power management device, so as to realize functions such as management of charging, discharging, and power consumption management through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The computer device 12 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described in detail here.

[0158] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to execute part of the method described in each embodiment of the present application.

[0159] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0160] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for three-dimensional reconstruction of anterior segment OCT images, characterized in that: The method comprises: Performing image semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence; Performing image registration on the semantic segmentation image sequence to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye; Extracting the target area contour from the anterior segment full circumference semantic segmentation image; Projecting the target area contour into three-dimensional space to obtain a point cloud of contours of various structures in the anterior segment of the eye; Based on the point cloud of the contours of the structures in the anterior segment of the eye, three-dimensional reconstruction is performed to obtain an anterior segment of the eye OCT three-dimensional image.

2. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 1, characterized in that: The performing image semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence includes: Performing image semantic segmentation on the anterior segment OCT image sequence through an OCT image semantic segmentation model based on U-Net or U2-Net to obtain a semantic segmentation image sequence; The downsampling operation of the U-Net or the U2-Net uses a convolution operation with a step size of 2.

3. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 1, characterized in that: The performing image registration on the semantic segmentation image sequence to obtain the semantic segmentation image of the entire periphery of the anterior segment of the eye comprises: Selecting an anterior segment OCT image from the anterior segment OCT image sequence as a reference OCT image, and selecting images other than the reference OCT image from the anterior segment OCT image sequence as OCT images to be registered; Calculate a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered; Performing pixel transformation on each semantic segmentation image in the semantic segmentation image sequence based on the rigid body transformation matrix to obtain a registered semantic segmentation image sequence; The semantic segmentation image sequence after the registration is merged to obtain a semantic segmentation image of the entire periphery of the anterior segment of the eye.

4. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 3, characterized in that: The calculating and obtaining a rigid body transformation matrix based on the reference OCT image and the OCT image to be registered comprises: Initialize the transformation parameters of the rigid body transformation matrix; In each iteration, a rigid body transformation is performed on the OCT image to be registered according to the current transformation parameters, and the similarity between the transformed image and the reference OCT image is calculated; Using a preset registration algorithm, updating transformation parameters to minimize the similarity; When the convergence conditions are met, the final calculated rigid body transformation matrix is ​​output.

5. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 1, characterized in that: The projecting of the target area contour into three-dimensional space to obtain the contour point cloud of each structure of the anterior segment of the eye includes: Get the angle value of each semantic segmentation image; Acquire a plurality of contour points and pixel coordinates of each contour point from the target area contour; According to the angle value and the pixel coordinates, the target area contour is projected into a cylindrical coordinate system to obtain a point cloud of contours of various structures of the anterior segment of the eye.

6. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 5, characterized in that: According to the angle value and the pixel coordinates, the target area contour is projected into a cylindrical coordinate system to obtain a point cloud of contours of various structures of the anterior segment of the eye, including: Determine a height value in the cylindrical coordinate system according to a first specified component coordinate in the pixel coordinate; Determine the radius value in the cylindrical coordinate system according to the second specified component coordinate in the pixel coordinate; Obtain three-dimensional coordinates according to the radius value, the angle value and the height value; Based on the three-dimensional coordinates, a point cloud is obtained.

7. The method for three-dimensional reconstruction of anterior segment OCT images according to claim 1, characterized in that: The method further comprises: Cache the anterior segment OCT image and the corresponding image identifier in a memory; An image reconstruction request is sent to Redis according to the image identifier, and an anterior segment OCT image corresponding to the image identifier and stored in the memory and returned by Redis is obtained.

8. A three-dimensional reconstruction device for anterior segment OCT images, characterized in that: The device comprises: A semantic segmentation module is used to perform semantic segmentation on the anterior segment OCT image sequence to obtain a semantic segmentation image sequence; An image registration module is used to perform image registration on the semantic segmentation image sequence to obtain an anterior segment full circumference semantic segmentation image; A contour extraction module, used to extract the target area contour from the anterior segment full circumference semantic segmentation image; A point cloud acquisition module is used to project the target area contour into three-dimensional space to obtain a point cloud of the contours of each structure of the anterior segment of the eye; The three-dimensional reconstruction module is used to perform three-dimensional reconstruction based on the contour point cloud of each structure of the anterior segment of the eye to obtain an anterior segment of the eye OCT three-dimensional image.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method for three-dimensional reconstruction of anterior segment OCT images as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for three-dimensional reconstruction of anterior segment OCT images as claimed in any one of claims 1 to 7 are implemented.

Images