Method and device for positioning fundus camera

Abstract

The invention discloses a fundus camera positioning method and device, and the method comprises the steps: obtaining a target eye image of a target object, and the target eye image at least comprises a target pupil of the target object; the target eye image is processed according to a pupil recognition model to determine pupil center point coordinates of a target pupil in the target eye image, and the pupil recognition model comprises a channel attention module and a space attention module which are used for processing the target eye image; and controlling the fundus camera to move to a target position corresponding to the pupil center point coordinate. Through the channel attention module and the space attention module, pupil recognition can be performed in combination with the image features of the target eye image in the channel dimension and the space dimension, so that the pupil recognition model more comprehensively pays attention to the important features of the target eye image in the multiple dimensions, the accuracy of the recognition result is effectively improved, and the recognition efficiency is improved. And the success rate of the fundus camera shooting the fundus image according to the pupil center point is improved.

Description

Technical Field

[0001] The present application relates to the technical field of fundus cameras, and in particular to a method and device for positioning a fundus camera. Background Art

[0002] Fundus camera is a medical imaging device used to photograph the inside of the eye, especially the retina and optic disc. It plays an important role in diagnosing and monitoring a variety of eye diseases, such as diabetic retinopathy, hypertensive retinopathy, macular degeneration, etc. During the operation of the fundus camera, the motor needs to move the fundus camera lens to a specific position relative to the center of the pupil. To do this, the center of the pupil needs to be identified.

[0003] The existing method for identifying the pupil center point will lose the accuracy of the pupil diameter during the pupil identification process, resulting in inaccurate identification of the pupil center point, thereby reducing the success rate of the fundus camera in capturing fundus images. Summary of the invention

[0004] To this end, this application discloses the following technical solutions:

[0005] The first aspect of the present application provides a method for positioning a fundus camera, comprising:

[0006] Obtaining a target eye image of a target object, wherein the target eye image at least includes a target pupil of the target object;

[0007] Processing the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, wherein the pupil recognition model includes a channel attention module and a spatial attention module for processing the target eye image;

[0008] According to the pupil center point coordinates and the pupil size, the fundus camera is controlled to move to a target position corresponding to the pupil center point coordinates.

[0009] Optionally, the method of obtaining the pupil recognition model includes:

[0010] Load the initial recognition model containing pre-trained weight parameters;

[0011] Processing the sample eye image according to the initial recognition model to obtain a labeling document corresponding to the sample eye image, wherein the labeling document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image;

[0012] Adjusting the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document until a recognition model whose model loss satisfies a convergence condition is obtained;

[0013] Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

[0014] Optionally, a method of obtaining the sample eye image includes:

[0015] Capturing and obtaining multiple frames of initial eye images;

[0016] Performing data enhancement processing on the initial eye image to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

[0017] Optionally, the pupil recognition model includes a target detection network, a channel attention module and a spatial attention module;

[0018] The step of processing the target eye image according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image includes:

[0019] extracting a first image feature of the target eye image according to the target detection network;

[0020] Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image;

[0021] Processing the fused channel attention features according to the spatial attention module to obtain a fused spatial attention feature of the target eye image;

[0022] The fused spatial attention features are processed according to the target detection network to obtain the pupil center point coordinates and pupil size of the target pupil in the target eye image.

[0023] Optionally, the target eye image includes two frames of target eye images captured by the fundus camera, and the fundus camera is a binocular synchronous fundus camera;

[0024] The controlling the fundus camera to move to a target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size includes:

[0025] Determine the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size of the two frames of the target eye images;

[0026] Determining a moving direction and a moving distance of the fundus camera according to the target position;

[0027] The stepping motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepping motor to move to the target position.

[0028] A second aspect of the present application provides a device for positioning a fundus camera, comprising:

[0029] an obtaining unit, configured to obtain a target eye image of a target object, wherein the target eye image at least includes a target pupil of the target object;

[0030] a processing unit, configured to process the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, wherein the pupil recognition model includes a channel attention module and a spatial attention module for processing the target eye image;

[0031] A control unit is used to control the fundus camera to move to a target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size.

[0032] Optionally, the device includes a training unit, which is used to obtain the pupil recognition model in the following manner:

[0033] Load the initial recognition model containing pre-trained weight parameters;

[0034] Processing the sample eye image according to the initial recognition model to obtain a labeling document corresponding to the sample eye image, wherein the labeling document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image;

[0035] Adjusting the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document until a recognition model whose model loss satisfies a convergence condition is obtained;

[0036] Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

[0037] Optionally, the training unit obtains the sample eye image in a manner including:

[0038] Capturing and obtaining multiple frames of initial eye images;

[0039] Performing data enhancement processing on the initial eye image to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

[0040] Optionally, the pupil recognition model includes a target detection network, a channel attention module and a spatial attention module;

[0041] When the processing unit processes the target eye image according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image, it is specifically used to:

[0042] extracting a first image feature of the target eye image according to the target detection network;

[0043] Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image;

[0044] Processing the fused channel attention features according to the spatial attention module to obtain a fused spatial attention feature of the target eye image;

[0045] The fused spatial attention features are processed according to the target detection network to obtain the pupil center point coordinates and pupil size of the target pupil in the target eye image.

[0046] Optionally, the target eye image includes two frames of target eye images captured by the fundus camera, and the fundus camera is a binocular synchronous fundus camera;

[0047] When the control unit controls the fundus camera to move to the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size, it is specifically used to:

[0048] Determine the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size of the two frames of the target eye images;

[0049] Determining a moving direction and a moving distance of the fundus camera according to the target position;

[0050] The stepping motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepping motor to move to the target position.

[0051] The beneficial effects of this solution are:

[0052] When identifying the pupil center coordinates of the target eye image based on the pupil recognition model, the target eye image is processed based on the channel attention module and the spatial attention module, so as to combine the image features of the target eye image in the channel and space dimensions to perform pupil center recognition, so that the pupil recognition model can pay more comprehensive attention to the important features of the target eye image in multiple dimensions, effectively improving the accuracy of the recognition result, and thereby improving the success rate of the fundus camera in taking fundus images according to the pupil center point. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 is a flow chart of a method for positioning a fundus camera provided in an embodiment of the present application;

[0055] Figure 2 It is a structural schematic diagram of a pupil recognition model provided in an embodiment of the present application;

[0056] Figure 3 It is a schematic diagram of the principle of a pupil positioning method provided in an embodiment of the present application;

[0057] Figure 4 is a schematic diagram of adjusting the position of the center point (point M) of a fundus camera provided in an embodiment of the present application;

[0058] Figure 5 It is a structural schematic diagram of a fundus camera positioning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] This application embodiment provides a method for positioning a fundus camera. Figure 1 , the method may include the following steps.

[0061] S101, obtaining a target eye image of a target object, where the target eye image at least includes a target pupil of the target object.

[0062] The target object may be any object for which fundus images need to be captured, for example, a patient with eye disease diagnosis needs such as diabetic retinopathy, hypertensive retinopathy, macular degeneration, etc. In step S101, the target eye of the target object may be captured by a binocular synchronous fundus camera to obtain two frames of target eye images of the target eye, and the target eye may be the left eye or the right eye of the target object.

[0063] Exemplarily, the captured target eye image can be as follows: Figure 3 shown in (1).

[0064] S102, processing the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, the pupil recognition model including a channel attention module and a spatial attention module for processing the target eye image.

[0065] In step S102, the two frames of target eye images captured can be respectively input into the pupil recognition model to identify the pupil center point coordinates and pupil size of the target pupil in each frame of the target eye image through the model, where the target pupil refers to the pupil of the captured target eye.

[0066] Specifically, the binocular synchronous fundus camera may include a left camera and a right camera, and the pupil recognition model may identify the pupil center point coordinates of the target eye image taken by the left camera, recorded as (Xl, Yl), which represents the position of the pupil center point of the target pupil in the frame image, and recognize the size of the target pupil of the image, which may specifically include the width Wl and height Hl of the target pupil in the image. Similarly, the pupil recognition model may identify the pupil center point coordinates of the target eye image taken by the right camera, recorded as (Xr, Yr), and recognize the size of the target pupil of the image, which are the width Wr and the height Hr, respectively.

[0067] S103, according to the pupil center coordinates and the pupil size, controlling the fundus camera to move to a target position corresponding to the pupil center coordinates.

[0068] In step S103, some inherent parameters of the fundus camera, such as the distance T between the left camera and the right camera, the focal length f of the fundus camera, etc., can be obtained from the relevant configuration information of the binocular synchronous fundus camera. Based on these parameters, the target position is determined in combination with the pupil center point coordinates and the pupil size, and then the movement of the fundus camera is controlled by a stepper motor used to drive the fundus camera so that the center point of the fundus camera moves to the target position corresponding to the pupil center point coordinates.

[0069] The beneficial effects of this embodiment are:

[0070] When identifying the pupil center coordinates of the target eye image based on the pupil recognition model, the target eye image is processed based on the channel attention module and the spatial attention module, so as to combine the image features of the target eye image in the channel and space dimensions to perform pupil center point recognition, so that the pupil recognition model can pay more comprehensive attention to the important features of the target eye image in multiple dimensions, effectively improving the accuracy of the recognition result, and thereby improving the success rate of the fundus camera in taking fundus images according to the pupil center point.

[0071] Optionally, the method of obtaining the pupil recognition model includes:

[0072] Load the initial recognition model containing pre-trained weight parameters;

[0073] Process the sample eye image according to the initial recognition model to obtain a label document corresponding to the sample eye image, wherein the label document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image;

[0074] Adjust the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document, until a recognition model whose model loss satisfies the convergence condition is obtained;

[0075] Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

[0076] The above-mentioned pre-trained weight parameters can be weight parameters contained in the image detection model obtained by training other images in the related field. Other images refer to images other than eye images. The advantage of loading pre-trained weight parameters is that the model performance can be improved through transfer learning methods.

[0077] After the initial recognition model is obtained, the sample eye images obtained can be inferred using the initial recognition model to obtain the annotation document corresponding to each image.

[0078] After obtaining the annotated document, the annotated document with errors can be corrected by manual correction, and the correct data can be added to the annotated document to obtain the corrected annotated document. For example, if the position of the center point of the pupil in a sample eye image is known to be (10, 20), and the position of the center point of the pupil in the sample eye image in the annotated document identified by the model is (20, 30), then it is determined that the annotated document has errors, and the (20, 30) of the annotated document can be modified to (10, 20) to obtain the corrected annotated document. Annotated documents without errors do not need to be corrected.

[0079] After obtaining the annotated documents and the corrected annotations, the model loss can be determined based on the deviation between the model's recognition results and the correct results. If the current model loss does not meet the convergence conditions, the weight parameters of the model can be adjusted according to the model loss to obtain a new model. If the current model loss meets the convergence conditions, the step of selecting the target recognition model can be executed.

[0080] The convergence condition can be that the model loss is less than or equal to a preset loss threshold.

[0081] The specific method of determining the model loss and adjusting the weight parameters according to the model loss can be found in the relevant prior art and will not be described in detail in this embodiment.

[0082] Optionally, when tuning model parameters, the relevant hyperparameters can be set to:

[0083] The initial value of the learning rate is set to 0.01, and the learning rate is adjusted by lr_scheduler. The optimizer is set to Adam, the optimizer momentum is set to 0.99, and the optimizer weight delay is set to 0.0005. The warmup epochs is set to 5, etc.

[0084] After each adjustment of the parameters to obtain a new model, the sample eye image can be processed again based on the new model to obtain the annotated document, and the process of adjusting the parameters according to the annotated document and the corrected annotated document can be iterated multiple times until the model loss meets the convergence condition.

[0085] In this embodiment, the model obtained after each parameter adjustment can be saved. Thus, when it is iterated multiple times until the model loss meets the convergence condition, multiple models obtained by multiple iterative processes can be obtained. When the model loss meets the convergence condition, an optimal target recognition model whose model evaluation indicators meet the evaluation conditions can be selected from the multiple models obtained by multiple iterative processes as the pupil recognition model used in this embodiment.

[0086] Model evaluation indicators may include any one or more of precision, recall, mAP, and intersection over union (IoU). The evaluation condition may be that the model evaluation indicator is greater than the corresponding indicator threshold, or it may be the model with the best evaluation indicator among multiple models, such as the highest precision or the highest recall.

[0087] Before using the initial recognition model to process the sample eye image data, any labeling tool (including but not limited to LabelMe, VIA, Label Studio, etc.) can be used to frame the pupil area in the sample eye image.

[0088] Optionally, after obtaining the trained pupil recognition model, in order to enable the model to better run on the ophthalmic detection device to which the fundus camera belongs, the trained model may be converted into a format adapted to the operating system of the ophthalmic detection device.

[0089] For example, if the ophthalmic detection device is based on the Android operating system, the trained model in pth format can be converted into a model in ncnn format, and the model can be deployed to the ophthalmic detection device based on the Android system through the ncnn tool; if the ophthalmic detection device runs the model based on GPU, no conversion is required, or the model can be converted into a trt model and then deployed on the GPU to accelerate GPU-based model reasoning; if the ophthalmic detection device runs the model based on CPU, the model can also be deployed to the CPU through openvino to accelerate CPU-based model reasoning.

[0090] Optionally, the method of obtaining a sample eye image includes:

[0091] Capturing and obtaining multiple frames of initial eye images;

[0092] The initial eye image is subjected to data enhancement processing to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

[0093] Optionally, in order to improve the generalization ability of the trained pupil recognition model, when taking multiple frames of initial eye images, a fundus camera can be used to collect multiple frames of eye images in different environments and among people of different age groups, thereby improving the generalization ability of the model by increasing the diversity of data.

[0094] The specific methods of the above-mentioned various data enhancement processing can be referred to the relevant existing technologies and will not be described in detail.

[0095] Data augmentation of the initial eye image helps improve the generalization ability of the trained pupil recognition model.

[0096] Optionally, the pupil recognition model of this embodiment includes a target detection network, a channel attention module and a spatial attention module. The target detection network can be a YOLOV9 network, that is, this embodiment can introduce channel attention and spatial attention mechanisms on the basis of the YOLOV9 network to obtain a pupil recognition model.

[0097] Based on the above model structure, the target eye image may be processed according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image in the following manner:

[0098] Extracting a first image feature of the target eye image according to the target detection network;

[0099] Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image;

[0100] Processing the fused channel attention features according to the spatial attention module to obtain the fused spatial attention features of the target eye image;

[0101] According to the target detection network processing and fusion of spatial attention features, the pupil center point coordinates and pupil size of the target pupil in the target eye image are obtained.

[0102] The method for extracting the first image feature by the target detection network can be referred to in the relevant prior art and will not be described in detail.

[0103] The way to obtain the fused channel attention features can be expressed by formulas (1) and (2).

[0104]

[0105]

[0106] F represents the first image feature, F1 represents the fused channel attention feature, Mc represents the channel attention module, and Mc(F) represents the channel attention feature obtained after the channel attention module processes the first image feature.

[0107] like Figure 2 As shown, after the first image feature F of C*H*W is obtained by the target detection network, F can be input into the channel attention module. After the channel attention module processes the first image feature F, a 1-dimensional channel attention feature Mc(F) can be obtained, that is, the dimension of Mc(F) is C*1*1.

[0108] Wherein, C represents the number of channels of the first image feature, H and W represent the height and width of the first image feature respectively, that is, the first image feature can be regarded as a set of C feature matrices with H rows and W columns.

[0109] The process of obtaining Mc(F) by the channel attention module can be that the channel attention module can include an average pooling layer AvgPool and a maximum pooling layer MaxPool. In the channel attention module, F can be input into these two pooling layers. After the average pooling layer processes F, the output is an average pooling feature map F with a dimension of C*1*1. avgc After the maximum pooling layer processes F, the output dimension is C*1*1 maximum pooling feature map F maxc Then, the two pooled feature maps are input into the multi-layer perceptron (MLP) neural network of the channel attention module, and the two features output after the MLP processes the two pooled feature maps are subjected to element-based addition operation. Finally, the result of the addition operation is activated based on the Sigmoid activation function to obtain the final channel attention feature Mc (F).

[0110] Wherein, W1 and W0 in formula (2) represent the weight parameter matrices contained in the MLP, wherein the dimension of W0 may be C / r*C, and the dimension of W1 may be C*C / r, where r is a preset reduction rate parameter.

[0111] After obtaining the channel attention features, such as Figure 2 As shown, the channel attention feature and the first image feature can be element-wise multiplied, and the result is the fused channel attention feature F1. Represents an element-wise multiplication operation.

[0112] After the fused channel attention feature F1 is processed by the spatial attention module, the fused spatial attention feature F2 can be obtained. The processing process can be expressed by the following formulas (3) and (4).

[0113]

[0114]

[0115] Among them, Ms represents the spatial attention module, and Ms(F1) represents the spatial attention features obtained after the spatial attention module processes F1.

[0116] Combination Figure 3 It can be seen from formula (3) that the way to obtain the fused spatial attention feature is that the spatial attention module can include an average pooling layer AvgPool and a maximum pooling layer MaxPool. In the spatial attention module, F1 can be input into the two pooling layers. After the average pooling layer processes F1, it outputs an average pooling feature map Favgs with a dimension of 1*H*W. After the maximum pooling layer processes F1, it outputs a maximum pooling feature map Fmaxs with a dimension of 1*H*W. Then, the two pooling feature maps are concatenated based on the channel. The dimension of the concatenated feature can be 2*H*W. Then, the concatenated feature is convolved with a 7*7 convolution layer. In addition, the convolution operation can be performed on the concatenated feature. In this way, the convolution feature with a dimension of 1*H*W is obtained after the dimension reduction and convolution processing. Finally, the convolution feature is processed by the Sigmoid activation function. The result after the Sigmoid activation function processing is the spatial attention feature Ms (F1). Ms(F1) can be a 2-dimensional feature with a dimension of 1*H*W, that is, the spatial attention feature is equivalent to a feature matrix with H rows and W columns.

[0117] Among them, [AvgPool (F1); MaxPool (F1)] means that the two pooling feature maps in the brackets are spliced ​​based on the channel, f 7*7 Represents the 7*7 convolutional layer in the spatial attention module. 7*7 ([AvgPool(F1); MaxPool(F1)]) means performing a 7*7 convolution operation on the channel-based concatenated features.

[0118] After obtaining the fused spatial attention features in the above manner, the fused spatial attention features can be processed by the module in the target detection network that is used to detect the pupil based on the image features to obtain the pupil center point coordinates and pupil size of the input image. The specific processing method can be referred to the relevant existing technology and will not be repeated here.

[0119] Optionally, the target eye image includes two frames of target eye images captured by a fundus camera, and the fundus camera is a binocular synchronous fundus camera;

[0120] According to the pupil center coordinates and pupil size, the fundus camera is controlled to move to the target position corresponding to the pupil center coordinates, including:

[0121] Determine the target position corresponding to the pupil center coordinates according to the pupil center coordinates and pupil sizes of the two frames of target eye images;

[0122] Determine the moving direction and moving distance of the fundus camera according to the target position;

[0123] The stepper motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepper motor to move to the target position.

[0124] The target position is determined as follows.

[0125] For the two frames of target eye images captured by the binocular synchronous fundus camera, the pupil center coordinates in the target eye image captured by the left camera can be marked as (Xl, Yl), and the pupil center coordinates in the target eye image captured by the right camera can be marked as (Xr, Yr). The fundus camera is a binocular synchronous fundus camera, the distance between the left camera and the right camera is T, and the focal lengths of the left camera and the right camera are recorded as f. T and f can be regarded as the inherent parameters of the binocular synchronous fundus camera.

[0126] In order to determine the target position, it is first necessary to determine the first coordinate system based on the binocular synchronous fundus camera and the eyes of the target object. The X-axis of the first coordinate system can be parallel to the line connecting the pupil centers of the two eyes of the target object, and the X-axis can be located on the focal plane of the binocular synchronous fundus camera (also called the front focal plane or the object focal plane), the Y-axis of the first coordinate system can be parallel to the symmetry axis of the face of the target object and also located on the focal plane, the Z-axis of the first coordinate system can be perpendicular to the focal plane of the binocular synchronous fundus camera and point to the eyes of the target object, and the Z-axis of the first coordinate system can pass through the position of the left camera, and the intersection of the Z-axis and the focal plane of the binocular synchronous fundus camera is the origin of the first coordinate system.

[0127] See also Figure 3 (1) and (2), assuming that point P is the actual pupil center. Figure 3 The real coordinates in the first coordinate system shown in (2) are denoted as (X测 , Y 测 , Z 测 ), it can be understood that the intersection of the line connecting point P and the left camera and the focal plane of the binocular synchronous fundus camera is the pupil center point of the target eye image captured by the left camera, and (Xl, Yl) is the X coordinate and Y coordinate of the intersection in the first coordinate system. The coordinate of point P relative to the left camera in the X-axis direction is Xl, and the coordinate of point P relative to the right camera in the X-axis direction is Xr. According to the similar triangle theorem, the following formulas (5) to (8) can be obtained.

[0128]

[0129]

[0130]

[0131]

[0132] Combining formula (5) and (6), we can obtain the following formula (9).

[0133]

[0134] Combining formulas (7) and (8), we can obtain the following formula (10).

[0135]

[0136] Combining formula (5) and (9), we can obtain the following formula (11).

[0137]

[0138] Where d represents the parallax of the binocular synchronous fundus camera, which is defined as d=Xl-Xr. 测 It can be understood as the depth information of the image, which is determined by the pupil size of the target eye image taken by the left camera, that is, the width Wl and height Hl of the pupil in the image, and the pupil size of the target eye image taken by the right camera, that is, the width Wr and height Hr of the pupil in the image.

[0139] It can be understood that for any point in space, as long as the coordinates of the point in the image taken by the binocular synchronous fundus camera are obtained, the coordinates of the point in space in the first coordinate system can be obtained based on the above formulas (5) to (11). The above calculation process is only an example of determining the true coordinates of the pupil center point based on the coordinates of the pupil center point in the target eye image.

[0140] The pupil center point P of the target eye determined in the above manner is the target position of this embodiment. The real coordinates (X测 , Y 测 , Z 测 ), which is the coordinate of the target position in the first coordinate system.

[0141] After the target position is determined, the moving direction and moving distance of the fundus camera can be determined as follows.

[0142] See also Figure 3 (2), in this embodiment, the center point M of the fundus camera refers to the intersection of a line drawn parallel to the Z axis through the midpoint of a line segment connecting the positions of the left camera and the right camera and the plane S, which is recorded as the center point M of the fundus camera. The plane S is parallel to the focal plane and contains point P.

[0143] Assume that the coordinates of point M in the first coordinate system are (X 目 , Y 目 , Z 目 ), according to the definition of point M above, we know that Z 目 and Z 测 are equal, so we only need to determine (X 目 , Y 目 ).

[0144] from Figure 3 It can be seen from (2) that from the Y-axis direction, point M is in the same plane as the left camera and the right camera, that is, the Y-axis coordinate of point M, the Y-axis coordinate of the left camera, and the Y-axis coordinate of the right camera are equal, that is, point M is in the same plane perpendicular to the Y-axis as the left camera and the right camera. The Z-axis direction can be understood as the depth information of the image. Among them, the coordinate of point M relative to the left camera can be recorded as (Xt, Yt), which is equivalent to the coordinate of point M in the image taken by the left camera. The coordinate can be determined according to the pupil size. The specific determination method can be referred to the relevant prior art, which will not be repeated in this embodiment.

[0145] Combining formulas (9) and (10), we can determine the coordinates of point M in the first coordinate system (X 目 , Y 目 ) and the coordinates of point M relative to the left camera have the relationship shown in formula (12).

[0146]

[0147] Therefore, after obtaining the coordinates of point M relative to the left camera, the coordinates of point M in the first coordinate system can be determined based on the coordinates.

[0148] Then, the moving direction and moving distance of the fundus camera may be determined according to the coordinates of point M in the first coordinate system and the coordinates of point P in the first coordinate system.

[0149] See also Figure 4 , Figure 4 A schematic diagram of adjusting the position of the center point of a fundus camera provided in an embodiment of the present application is shown.

[0150] like Figure 4 As shown, Figure 4 The circles from (1) to (9) represent the imaging plane of the target eye image (i.e., the aforementioned Figure 3 The horizontal dotted line represents the Y-axis direction, and the vertical dotted line represents the X-axis direction, where the Y-axis direction is Figure 3 It can be understood as the height coordinate information. The dotted intersection point in the middle of each circle represents the M point mentioned above (X 目 , Y 目 ), each plus sign "+" in the circle represents the coordinates of the actual pupil center point P in the first coordinate system (X 测 , Y 测 ). In this embodiment, the binocular synchronous fundus camera can be installed on an ophthalmic detection device, and the ophthalmic detection device can be equipped with a three-axis drive module. The three-axis drive module can include multiple stepper motors. The three-axis drive module can control the binocular synchronous fundus camera to move on the X-axis, Y-axis, and Z-axis through these stepper motors, so that the center point M of the fundus camera coincides with the actual pupil center point P, that is, the center point M of the fundus camera is moved to the position where the actual pupil center point P is located.

[0151] Specifically, the distance to be moved in the X-axis direction can be recorded as X 移 , then the distance required to move in the X-axis direction can be calculated according to the following formula (13).

[0152]

[0153] Since the actual pupil center point P may be located on the left and right sides of point M, the movement direction of the X-axis needs to be determined according to the relative positions of the actual pupil center point P and point M.

[0154] like Figure 4 As shown in (3), (6) and (9), point M is located to the left of the actual pupil center point P, that is, Xl is greater than Xt. Therefore, it is necessary to control the three-axis drive module to move the fundus camera to the right along the X axis by a moving distance X. 移 =Z 测 / f (Xl-Xt).

[0155] like Figure 4 As shown in (2), (5) and (8), the X-axis coordinates of point M and the actual pupil center point P are the same, that is, X 目 =X 测 , so there is no need to move the fundus camera along the X axis, X 移 =0.

[0156] like Figure 4 As shown in (1), (4) and (7), point M is located to the right of the actual pupil center point P, that is, Xl is less than Xt. Therefore, it is necessary to control the three-axis drive module to move the fundus camera to the left along the X axis by a moving distance X. 移 =Z 测 / f (Xt-Xl).

[0157] Similarly, the distance to be moved in the Y-axis direction can be recorded as Y 移 .

[0158] like Figure 4 As shown in (1), (2) and (3), point M is located below the actual pupil center point P, that is, Yl is greater than Yt. Therefore, it is necessary to control the three-axis drive module to move the fundus camera upward along the Y axis by a distance Y. 移 =Z 测 / f (Yl-Yt).

[0159] like Figure 4 As shown in (4), (5) and (6), the Y-axis coordinates of point M and the actual pupil center point P are the same, that is, Y 目 =Y 测 , so there is no need to move the fundus camera along the Y axis, Y 移 =0.

[0160] like Figure 4 As shown in (7), (8) and (9), point M is located above the actual pupil center point P, that is, Yl is smaller than Yt. Therefore, it is necessary to control the three-axis drive module to move the fundus camera downward along the Y axis by a distance Y. 移 =Z 测 / f (Yt-Yl).

[0161] It should be noted that in the pupil positioning method provided in the embodiment of the present application, the X-axis direction can be understood as the coordinate information of the left camera and the right camera extending horizontally, the Y-axis direction can be understood as the height information of the plane where the left camera and the right camera are located, and the Z-axis direction can be understood as the depth information of the image. Therefore, it is necessary to move the center point of the fundus camera to the actual center point of the pupil.

[0162] After the fundus camera is controlled to move in the above manner, the point M corresponding to the fundus camera can coincide with the pupil center point P of the photographed target eye.

[0163] The present application embodiment provides a device for positioning a fundus camera, see Figure 5 , the device may include the following units.

[0164] An obtaining unit 501 is used to obtain a target eye image of a target object, where the target eye image at least includes a target pupil of the target object;

[0165] A processing unit 502 is used to process the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, wherein the pupil recognition model includes a channel attention module and a spatial attention module for processing the target eye image;

[0166] The control unit 503 is used to control the fundus camera to move to a target position corresponding to the pupil center coordinates according to the pupil center coordinates and the pupil size.

[0167] Optionally, the apparatus includes a training unit 504, configured to obtain a pupil recognition model in the following manner:

[0168] Load the initial recognition model containing pre-trained weight parameters;

[0169] Process the sample eye image according to the initial recognition model to obtain a label document corresponding to the sample eye image, wherein the label document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image;

[0170] Adjust the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document, until a recognition model whose model loss satisfies the convergence condition is obtained;

[0171] Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

[0172] Optionally, the training unit 504 obtains the sample eye image in the following manner:

[0173] Capturing and obtaining multiple frames of initial eye images;

[0174] The initial eye image is subjected to data enhancement processing to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

[0175] Optionally, the pupil recognition model includes a target detection network, a channel attention module and a spatial attention module;

[0176] When the processing unit 502 processes the target eye image according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image, it is specifically used to:

[0177] Extracting a first image feature of the target eye image according to the target detection network;

[0178] Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image;

[0179] Processing the fused channel attention features according to the spatial attention module to obtain the fused spatial attention features of the target eye image;

[0180] According to the target detection network processing and fusion of spatial attention features, the pupil center point coordinates and pupil size of the target pupil in the target eye image are obtained.

[0181] Optionally, the target eye image includes two frames of target eye images captured by a fundus camera, and the fundus camera is a binocular synchronous fundus camera;

[0182] When the control unit 503 controls the fundus camera to move to the target position corresponding to the pupil center coordinates according to the pupil center coordinates and the pupil size, it is specifically used to:

[0183] Determine the target position corresponding to the pupil center coordinates according to the pupil center coordinates and pupil sizes of the two frames of target eye images;

[0184] Determine the moving direction and moving distance of the fundus camera according to the target position;

[0185] The stepper motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepper motor to move to the target position.

[0186] The working principle of the fundus camera positioning device provided in this embodiment can be found in the relevant steps of the fundus camera positioning method provided in any embodiment of the present application, and will not be repeated here.

[0187] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0188] For the convenience of description, the above system or device is described by dividing it into various modules or units according to its functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0189] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0190] Finally, it should be noted that, in this article, relational terms such as first, second, third and fourth are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0191] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for positioning a fundus camera, characterized in that: include: Obtaining a target eye image of a target object, wherein the target eye image at least includes a target pupil of the target object; Processing the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, wherein the pupil recognition model includes a channel attention module and a spatial attention module for processing the target eye image; According to the pupil center point coordinates and the pupil size, the fundus camera is controlled to move to a target position corresponding to the pupil center point coordinates.

2. The method according to claim 1, characterized in that The method of obtaining the pupil recognition model includes: Load the initial recognition model containing pre-trained weight parameters; Processing the sample eye image according to the initial recognition model to obtain a labeling document corresponding to the sample eye image, wherein the labeling document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image; Adjusting the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document until a recognition model whose model loss satisfies a convergence condition is obtained; Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

3. The method according to claim 2, characterized in that The method of obtaining the sample eye image includes: Capturing and obtaining multiple frames of initial eye images; Performing data enhancement processing on the initial eye image to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

4. The method according to claim 1, characterized in that The pupil recognition model includes a target detection network, a channel attention module and a spatial attention module; The step of processing the target eye image according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image includes: extracting a first image feature of the target eye image according to the target detection network; Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image; Processing the fused channel attention features according to the spatial attention module to obtain a fused spatial attention feature of the target eye image; The fused spatial attention features are processed according to the target detection network to obtain the pupil center point coordinates and pupil size of the target pupil in the target eye image.

5. The method according to claim 1, characterized in that The target eye image includes two frames of target eye images captured by the fundus camera, and the fundus camera is a binocular synchronous fundus camera; The controlling the fundus camera to move to a target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size includes: Determine the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size of the two frames of the target eye images; Determining a moving direction and a moving distance of the fundus camera according to the target position; The stepping motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepping motor to move to the target position.

6. A device for positioning a fundus camera, characterized in that: include: an obtaining unit, configured to obtain a target eye image of a target object, wherein the target eye image at least includes a target pupil of the target object; a processing unit, configured to process the target eye image according to a pre-acquired pupil recognition model to determine the pupil center point coordinates and pupil size of the target pupil in the target eye image, wherein the pupil recognition model includes a channel attention module and a spatial attention module for processing the target eye image; A control unit is used to control the fundus camera to move to a target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size.

7. The device according to claim 6, characterized in that The apparatus comprises a training unit for obtaining the pupil recognition model in the following manner: Load the initial recognition model containing pre-trained weight parameters; Processing the sample eye image according to the initial recognition model to obtain a labeling document corresponding to the sample eye image, wherein the labeling document represents the coordinates of the pupil center point and the length and width of the pupil in the sample eye image; Adjusting the weight parameters contained in the initial recognition model according to the annotated document and the corrected annotated document corresponding to the annotated document until a recognition model whose model loss satisfies a convergence condition is obtained; Among the multiple recognition models obtained by adjusting the weight parameters, a target recognition model is selected as the pupil recognition model according to the model evaluation index.

8. The device according to claim 7, characterized in that The training unit obtains the sample eye image in the following manner: Capturing and obtaining multiple frames of initial eye images; Performing data enhancement processing on the initial eye image to obtain multiple frames of sample eye images, wherein the data enhancement processing includes at least one of scaling, rotating, flipping, blurring, and mixing.

9. The device according to claim 6, characterized in that The pupil recognition model includes a target detection network, a channel attention module and a spatial attention module; When the processing unit processes the target eye image according to the pre-acquired pupil recognition model to determine the pupil center point coordinates of the target pupil in the target eye image, it is specifically used to: extracting a first image feature of the target eye image according to the target detection network; Processing the first image feature according to the channel attention module to obtain a fused channel attention feature of the target eye image; Processing the fused channel attention features according to the spatial attention module to obtain a fused spatial attention feature of the target eye image; The fused spatial attention features are processed according to the target detection network to obtain the pupil center point coordinates and pupil size of the target pupil in the target eye image.

10. The device according to claim 6, characterized in that The target eye image includes two frames of target eye images captured by the fundus camera, and the fundus camera is a binocular synchronous fundus camera; When the control unit controls the fundus camera to move to the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size, it is specifically used to: Determine the target position corresponding to the pupil center point coordinates according to the pupil center point coordinates and the pupil size of the two frames of the target eye images; Determining a moving direction and a moving distance of the fundus camera according to the target position; The stepping motor is controlled according to the moving direction and the moving distance, so that the center point of the fundus camera is driven by the stepping motor to move to the target position.

Images