Eye injection point detection method and controller, ophthalmic robot and storage medium

Abstract

The embodiment of the invention provides an eye injection point detection method, a controller, an ophthalmology robot and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a sample initial eye image belonging to a first region from a preset image data set; acquiring a color difference threshold value; performing color difference processing on the sample initial eye image according to a color difference threshold to obtain a sample target eye image belonging to the second region; training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model; acquiring an original eye image belonging to the second area; performing key point identification on the original eye image through a target eye image detection model to obtain pupil key points, iris key points and canthus key points; according to the pupil key points, the iris key points and the canthus key points, injection point detection is carried out, and a target injection point is obtained. According to the embodiment of the invention, the detection precision and universality of the human eye key point detection method can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an eye injection point detection method and controller, an ophthalmic robot and a storage medium. Background Art

[0002] Human eye key point detection is a technology that detects facial images or videos to identify the key points of the human eye (such as the corners of the eyes, pupils, and irises). Human eye key point detection can be applied to scenarios such as ophthalmic diagnosis (such as eye injections, ophthalmic robots). However, the current human eye key point detection methods are difficult to meet the requirements of eye injections for detection accuracy. In addition, human eye key point detection methods often target the eyes of users in specific areas, and there are differences in eyes in different areas, and they cannot be well applied to user eyes in other areas outside of specific areas. It can be seen that the detection accuracy of the human eye key point detection method is low and its scope of application is narrow.

[0003] Therefore, how to improve the detection accuracy and universality of the human eye key point detection method has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an eye injection point detection method and controller, an ophthalmic robot and a storage medium, aiming to improve the detection accuracy and universality of the human eye key point detection method, which can be applied to ophthalmic robots.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application provides a method for detecting an eye injection point, the method comprising:

[0006] Acquire a sample initial eye image from a preset image data set; the eye images in the image data set all belong to the first area;

[0007] Acquire a color difference threshold; the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second area, and the first area and the second area are different areas;

[0008] Performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area;

[0009] Training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model;

[0010] Acquire an original eye image belonging to the second area;

[0011] Using the target eye image detection model, key points of the original eye image are identified to obtain pupil key points, iris key points and eye corner key points;

[0012] Injection point detection is performed based on the pupil key point, the iris key point and the eye corner key point to obtain the target injection point.

[0013] In some embodiments, the sample initial eye image includes a pupil area image and an initial iris area image; performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area includes:

[0014] Performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold;

[0015] Performing image synthesis on the migrated iris region image and the initial iris region image to obtain a target iris pupil region image;

[0016] The sample initial eye image is updated according to the target iris pupil area image to obtain the sample target eye image belonging to the second area.

[0017] In some embodiments, performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image includes:

[0018] Performing histogram statistics based on the brightness value of the pupil area image to obtain a pupil area histogram;

[0019] Performing histogram statistics based on the brightness value of the initial iris region image to obtain an iris region histogram;

[0020] Performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship;

[0021] The brightness value of the initial iris region image is updated according to the color difference threshold and the brightness mapping relationship to obtain the migrated iris region image.

[0022] In some embodiments, synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image includes:

[0023] Obtaining a weight of the initial iris region image to obtain a first weight;

[0024] Obtaining the weight of the migrated iris region image to obtain a second weight, wherein the sum of the first weight and the second weight is 1;

[0025] The target iris pupil area image is obtained by weighted fusion based on the first weight, the initial iris area image, the second weight and the migrated iris area image.

[0026] In some embodiments, the step of performing key point recognition on the original eye image by using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points includes:

[0027] Normalizing the original eye image to obtain a standard eye image;

[0028] Marking predicted key points on the standard eye image;

[0029] Performing image segmentation on the standard eye image according to the predicted key points to obtain a pupil key area, an iris key area and an eye corner key area;

[0030] Extracting key points from the pupil key area to obtain the pupil key points;

[0031] Extracting key points from the iris key area to obtain the iris key points;

[0032] Key points are extracted from the eye corner key area to obtain the eye corner key points.

[0033] In some embodiments, the step of performing injection point detection according to the pupil key point, the iris key point, and the eye corner key point to obtain the target injection point includes:

[0034] Performing ellipse fitting on the iris key points to obtain an iris edge ellipse;

[0035] Obtaining a first eye line segment according to the pupil key point and the eye corner key point;

[0036] Obtaining an intersection point between the first eye line segment and the iris edge ellipse as an eye reference point;

[0037] Point selection is performed on the first eye line segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

[0038] In some embodiments, obtaining a first eye line segment according to the pupil key point and the eye corner key point includes:

[0039] The pupil center point is calculated according to the pupil key point;

[0040] According to the position of the eye corner key point and the preset position constraint condition, selecting the eye corner edge point from the plurality of eye corner key points;

[0041] The pupil center point and the eye corner edge point are connected to obtain the first eye line segment.

[0042] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes a controller, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0043] To achieve the above object, the third aspect of the embodiment of the present application proposes an ophthalmic robot, the ophthalmic robot comprising a robotic arm, a collection module and the controller described in the second aspect above:

[0044] The controller is electrically connected to the acquisition module and the robotic arm:

[0045] The acquisition module is used to acquire original eye images, and the robotic arm is used to perform injection operations according to the target injection point.

[0046] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0047] The eye injection point detection method and controller, ophthalmic robot and storage medium proposed in the present application obtain a sample initial eye image belonging to the first area from a preset image data set; obtain a color difference threshold, the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second area; then perform color difference processing based on the color difference threshold and the sample initial eye image to obtain a sample target eye image belonging to the second area; since the sample target eye image obtained after processing belongs to the second area, the target eye image detection model obtained after training the preset initial eye image detection model with the sample target eye image can accurately identify the eye key points for the eyeball of the second area, avoiding the recognition accuracy of the model being affected by the area to which the image belongs in the image data set, thereby avoiding the obvious realization of eye key point detection only for the image of the first area. In addition, there may be differences in the eyeballs of objects in different areas. In the embodiment of the present application, the color difference threshold can be adjusted based on the corresponding different areas, and the universality is high. For the original eye image belonging to the second area, the trained target eye image detection model can accurately identify the pupil key points, iris key points and eye corner key points, and then perform injection point detection, and the detected target injection points are more accurate. The embodiment of the present application can improve the detection accuracy and universality of the human eye key point detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of an eye injection point detection method provided in an embodiment of the present application;

[0049] Figure 2 This is provided by an embodiment of the present application Figure 1 Flow chart of step 103 in;

[0050] Figure 3 is a schematic diagram of a sample initial eye image provided by an embodiment of the present application;

[0051] Figure 4 This is provided by an embodiment of the present application Figure 2 Flow chart of step 201 in;

[0052] Figure 5 is a schematic diagram of a pupil area image provided by an embodiment of the present application;

[0053] Figure 6 is a schematic diagram of an initial iris region image provided by an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of an iris region image after migration provided by an embodiment of the present application;

[0055] Figure 8 is a schematic diagram of a target iris pupil area image provided by an embodiment of the present application;

[0056] Fig. 9 is a schematic diagram of a target injection point provided by an embodiment of the present application;

[0057] Fig.10 It is a schematic diagram of the hardware structure of the controller provided in the embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] First, some nouns involved in this application are analyzed:

[0062] Eye injection: also known as intraocular administration, is a treatment method that directly injects drugs into the eye. It can be used in the treatment of a variety of eye diseases. Eye injection includes vitreous cavity injection, subretinal space injection, and suprachoroidal space injection.

[0063] Ophthalmic robot: An ophthalmic robot is a surgical robot used in eye surgery.

[0064] Artificial Intelligence (AI): is a technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial Intelligence includes image recognition, robotics, neural networks, deep learning, etc.

[0065] Image recognition: refers to the technology of using computers to process, analyze and understand images to identify targets and objects of various different patterns. Image recognition is a practical application of deep learning algorithms.

[0066] Neural Network (NN): It is a key technology in the field of artificial intelligence. Specifically, it can be a mathematical model or computational model that imitates the structure and function of biological neural networks. Convolutional Neural Networks (CNN) is one of the representative algorithms of deep learning.

[0067] Rectified Linear Unit (ReLU): is a nonlinear activation function commonly used in neural networks, which can enhance the nonlinear characteristics of neural networks.

[0068] Batch Normalization (BN): refers to the normalization operation on data. It is a technology used to improve the performance and stability of neural networks. It can suppress the gradient disappearance or gradient explosion of deep neural networks.

[0069] Transformer model: is a deep learning model based on the self-attention mechanism that can be used to process sequence data. The self-attention mechanism, also known as the internal attention mechanism, is an attention mechanism that associates different positions of a single sequence to calculate the representation of the same sequence.

[0070] The eye injection point detection method and controller, ophthalmic robot and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the eye injection point detection method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0072] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0073] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 is an optional flow chart of the eye injection point detection method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps 101 to 107.

[0075] Step 101, obtaining a sample initial eye image from a preset image data set; the eye images in the image data set all belong to the first area;

[0076] Step 102, obtaining a color difference threshold; the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second area, and the first area and the second area are different areas;

[0077] Step 103, performing color difference processing on the sample initial eye image according to the color difference threshold value to obtain a sample target eye image belonging to the second area;

[0078] Step 104, training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model;

[0079] Step 105, obtaining an original eye image belonging to the second area;

[0080] Step 106, performing key point recognition on the original eye image through the target eye image detection model to obtain pupil key points, iris key points and eye corner key points;

[0081] Step 107, performing injection point detection based on pupil key points, iris key points and eye corner key points to obtain the target injection point.

[0082] The beneficial effects of the embodiments of the present application include but are not limited to: the sample target eye image obtained after the color difference processing belongs to the second area, so the target eye image detection model obtained after training the preset initial eye image detection model with the sample target eye image can accurately identify the eye key points for the eyeballs in the second area, avoiding the recognition accuracy of the model being affected by the area to which the image belongs in the image data set, thereby avoiding the model being able to only detect the eye key points for the image in the first area; the eyeballs of objects in different areas may be different, so the color difference threshold can be adjusted based on the corresponding different areas in the embodiments of the present application, and the universality is high. For the original eye image belonging to the second area, the trained target eye image detection model can accurately identify the pupil key points, iris key points and eye corner key points, and then perform injection point detection, so the detected target injection points are also more accurate. It can be seen that the embodiments of the present application can improve the detection accuracy and universality of the human eye key point detection method.

[0083] In step 101 of some embodiments, the data types in the image data set may include images and videos. Obtaining a sample initial eye image from a preset image data set may be to obtain a video from the image data set, and sampling the video according to a preset sampling interval to obtain a sample initial eye image. Specifically, the video size may be about 200G, and the sampling interval may be to capture an image every two seconds. By setting a suitable sampling interval, repeated samples can be minimized, and video frames with large differences in the video can be extracted as training samples for the initial eye image detection model, thereby improving the reliability of the training model. The sample initial eye image may also be obtained in other ways, but is not limited to this.

[0084] It should be noted that the color difference refers to the color difference between the pupil and the iris in the eye image. Specifically, the eye images (including the above-mentioned sample initial eye images, sample target eye images, original eye images, etc.) are all grayscale images. In this case, the color difference refers to the absolute value of the difference in grayscale values ​​between the pupil area and the iris area in the eye image. The larger the color difference, the greater the brightness difference between the pupil area and the iris area.

[0085] It should be noted that the color difference threshold represents the maximum difference between the pupil and the iris of the eye image belonging to the second area.

[0086] In step 102 of some embodiments, the first area and the second area are different areas, and the eyes of users in different areas may have large differences in iris color, iris and pupil color difference, etc. Such differences often cause the model trained using eye images belonging to a specific area to be unable to accurately detect key points of eye images belonging to other areas, and may even cause the trained model to be unsuitable for eye images in other areas. In an embodiment of the present application, the trained model needs to perform key point detection on eye images belonging to the second area, while the eye images in the data set all belong to the first area, and recollecting image data requires a lot of manpower and time. In view of the above situation, a color difference threshold is obtained, and the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second area; then, in step 103, the sample initial eye image is subjected to color difference processing according to the color difference threshold to obtain a sample target eye image.

[0087] In step 103 of some embodiments, the sample target eye image belongs to the second region, which means that the color difference of the sample target eye image is less than the color difference threshold, that is, the sample target eye image meets the eye color difference characteristics of the second region.

[0088] In step 104 of some embodiments, multiple sample initial eye images may be obtained from the image data set and processed separately to obtain multiple sample target eye images, and the preset initial eye image detection model may be trained with the multiple sample target eye images to obtain a target eye image detection model. By obtaining a large number of samples to train the model, the detection accuracy of the trained target eye image detection model for the eye pictures belonging to the second area in detecting eye key points can be improved, thereby improving the accuracy of the target injection point obtained based on the subsequent key point detection.

[0089] In step 105 of some embodiments, the original eye image is not labeled. The original eye image can be obtained by an image acquisition device, for example, by photographing the eyes of the user belonging to the second area through a camera, so as to obtain the original eye image. Specifically, a camera with a field of view (FOV) of 30° and a focal length of 10 cm can be used. In order to ensure a stable light source, a dimmable circular fill light aperture can also be used during shooting to ensure clear eye imaging during shooting.

[0090] In step 106 of some embodiments, the target eye image detection model may be a Transformer model, which is a deep learning model based on a self-attention mechanism.

[0091] In step 107 of some embodiments, after determining the target injection point, the target injection point can provide a reference for some scenarios that require drug injection. For example, in some scenarios, a syringe can be used to inject drugs at the position of the eyeball corresponding to the target injection point. It should be noted that ocular drug injection is a treatment method that directly injects drugs into the inside of the eye. The embodiments of the present application are mainly aimed at vitreous cavity injection, which means injecting drugs directly into the vitreous cavity of the eye. In addition, this method can be applied to ophthalmic robots. At present, drug injections are often performed manually, which may cause damage to the eyeball due to hand shaking during injection. Compared with traditional manual injection operations, ophthalmic robots have higher control accuracy, which can improve injection accuracy, thereby reducing damage to the eyeball caused by injection.

[0092] See also Figure 2 In some embodiments, the sample initial eye image includes a pupil area image and an initial iris area image. Step 103 may include but is not limited to steps 201 to 203:

[0093] Step 201, performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold;

[0094] Step 202, synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image;

[0095] Step 203, updating the sample initial eye image according to the target iris pupil area image to obtain a sample target eye image belonging to the second area.

[0096] The advantage of this embodiment is that the color of the initial iris region image is migrated according to the color difference threshold and the pupil region image to obtain the migrated iris region image, so that the color difference between the migrated iris and pupil is less than the color difference threshold, and the color difference between the iris and pupil in the eye image meets the constraint of the color difference threshold, so that the synthesized target iris pupil region image is closer to the color difference feature of the second region. The sample initial eye image is updated according to the target iris pupil region image to obtain a sample target eye image belonging to the second region, so that the model can be trained with the samples belonging to the second region, so that the model can accurately detect eye key points of the eye image belonging to the second region, thereby improving the detection accuracy and universality of the human eye key point detection method.

[0097] Specifically, before step 201, the method may include: performing image segmentation on the sample initial eye image through a preset initial eye image detection model to obtain a pupil area image and an iris area image. In some embodiments, the eye images (including the sample initial eye images) in the image data set are all labeled, and the labels may be manually labeled or predicted by a model, which is not limited in the embodiments of the present application. The form of the label may include text annotation and video annotation. Specifically, the sample initial eye image is provided with a segmentation label and a key point prediction label. The segmentation label is used to mark the eye corner area, iris area and pupil area of ​​the eye image, and the key point prediction label is used to mark the eye corner prediction key points, iris prediction key points and pupil prediction key points of the eye image. The segmentation label can improve the accuracy of the above-mentioned image segmentation, and the joint point prediction label can improve the efficiency of key point detection of the sample target eye image.

[0098] See also Figure 3 In some embodiments, the sample initial eye image has segmentation labels, key point prediction labels, and viewpoint estimation labels. Figure 3 As shown in the figure, each row is the same image and its corresponding various labels. The first column from the left is the original image, such as the grayscale image acquired by the camera. The second column is the segmentation label, where the red area is the eye corner area, the green area is the iris area, and the blue area is the pupil area. The third column is the key point prediction label, where the red point is the eye corner prediction key point, the green point is the iris prediction key point, and the white point is the pupil prediction key point. The fourth column is the viewpoint estimation label, the green circle is the eyeball prediction range, and the line segment from the pupil to the center of the circle is used to represent the user's viewpoint.

[0099] In step 201 of some embodiments, color migration of the initial iris area image refers to updating the grayscale value of the initial iris area image to obtain a migrated iris area image, so as to update the color difference between the migrated iris area image and the pupil area image.

[0100] In step 202 of some embodiments, the migrated iris area image and the initial iris area image are synthesized by averaging synthesis, weighted synthesis, etc., which is not limited in the embodiments of the present application.

[0101] In step 203 of some embodiments, updating the sample initial eye image according to the target iris pupil area image means updating the iris area and pupil area in the sample initial eye image with the target iris pupil area image to obtain a new complete eye image, i.e., the sample target eye image, which includes other information such as the corner of the eye area.

[0102] For example, the pupil area image, the initial iris area image, and the iris area image after migration are all grayscale images, and the color difference refers to the absolute value of the difference between the grayscale values ​​of the pixels in the image. The grayscale value ranges from 0 to 255, where 0 represents black and 255 represents white. For the eyeball belonging to the first area, the pupil is black and the iris is light blue; for the eyeball belonging to the second area, the pupil is black and the iris is tan. In some embodiments, red (R), green (G), and blue (B) are used to represent colors, black is represented by RGB (0,0,0), light blue is represented by RGB (0,204,255), and tan is represented by RGB (255,102,0), and the grayscale value is calculated as (R+G+B) / 3. Then the grayscale value corresponding to black is 0, the grayscale value corresponding to light blue is 153, and the grayscale value corresponding to tan is 119. The sample initial eye image belonging to the first region includes a pupil region image and an initial iris region image. The grayscale value of the pupil region image is 0, and the grayscale value of the initial iris region image is 153. Therefore, the color difference of the sample initial eye image is 153-0=153. The grayscale value of the migrated iris region image is 119. Assuming that the migrated iris region image and the initial iris region image are averaged to obtain the target iris pupil region image, the iris region grayscale value in the target iris pupil region image is (119+153) / 2=136, and the color difference of the target iris pupil region image is 136-0=136. Assuming that the color difference threshold is 140, it can be seen that the color difference 153 of the sample initial eye image is greater than the color difference threshold, while the color difference 136 of the target iris pupil region image is less than the color difference threshold. The color difference threshold represents the maximum difference between the pupil and the iris of the eye image belonging to the second region. Therefore, the target iris pupil region belongs to the second region. In the embodiment of the present application, the sample initial eye image is updated by the target iris pupil area image, and the obtained sample target eye image also belongs to the second area. Therefore, training the model with the sample target eye image can improve the detection accuracy of the model for eye key point detection of the eye image belonging to the second area. The user can also adjust the color difference threshold for the area to which the eye image to be detected belongs, so this method improves the universality of the detection method.

[0103] See also Figure 4 In some embodiments, step 201 may include but is not limited to steps 401 to 404:

[0104] Step 401, performing histogram statistics based on the brightness value of the pupil area image to obtain a pupil area histogram;

[0105] Step 402, performing histogram statistics based on the brightness values ​​of the initial iris region image to obtain an iris region histogram;

[0106] Step 403, performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship;

[0107] Step 404: update the brightness value of the initial iris region image according to the color difference threshold and the brightness mapping relationship to obtain a migrated iris region image.

[0108] The advantage of this embodiment is that the brightness values ​​of the pupil area image and the initial iris area image are represented by a histogram, and the pupil area histogram and the iris area histogram are matched to obtain a brightness mapping relationship;

[0109] It should be noted that, since the images are all grayscale images, the brightness value in the embodiments of the present application is equivalent to the grayscale value of the image.

[0110] It should be noted that histogram statistics refers to statistically analyzing the frequency distribution of each gray level in an image to form a gray level histogram.

[0111] In step 403 of some embodiments, histogram matching refers to making the grayscale distribution of the initial iris region image similar to the grayscale distribution of the pupil region image based on the pupil region histogram to obtain a brightness mapping relationship.

[0112] In step 404 of some embodiments, the brightness mapping relationship may be a grayscale transformation relationship. Specifically, the brightness mapping relationship is a discontinuous mapping relationship, for example, pixels with a grayscale value of 100 in the initial iris region image are converted into pixels with a grayscale value of 140, pixels with a grayscale value of 50 are converted into pixels with a grayscale value of 80, and so on, thereby obtaining the migrated iris region image. The brightness mapping relationship may be obtained by methods such as statistical historical data, and the embodiments of the present application do not limit this.

[0113] In some embodiments, step 202 includes:

[0114] Obtaining the weight of the initial iris region image to obtain a first weight;

[0115] Obtaining the weight of the migrated iris region image to obtain a second weight; the sum of the first weight and the second weight is 1;

[0116] A weighted fusion is performed according to the first weight, the initial iris region image, the second weight and the migrated iris region image to obtain a target iris pupil region image.

[0117] The advantage of this embodiment is that by obtaining the weight of the initial iris area image and the weight of the migrated iris area image, the images are weightedly fused to obtain a target iris pupil area image, so that the color difference of the target iris pupil area image can be adjusted by the first weight and the second weight, so that the color difference between the pupil and the iris in the target iris pupil area image is closer to the color difference of the eye image belonging to the second area, thereby improving the flexibility of the method.

[0118] It should be noted that the iris region in the eye image includes the pupil region, and the fused target iris-pupil region image includes the iris and the pupil.

[0119] It should be noted that the first weight is used to adjust the proportion of the initial iris region image in the weighted fusion process, and the second weight is used to adjust the proportion of the migrated iris region image in the weighted fusion process.

[0120] In some embodiments, the target iris pupil area image is determined by the following formula:

[0121] S×FM1+(1-S)×FM2=FM3,

[0122] Wherein, S represents the first weight, (1-S) represents the second weight, FM1 represents the initial iris region image, FM2 represents the migrated iris region image, and FM3 represents the target iris pupil region image.

[0123] Specifically, the value range of S may be [0, 1], for example, S = 0.5. If S = 0, it means that the migrated iris region image is completely used as the target iris pupil region image.

[0124] See also Figures 5 to 8 In some embodiments, the pupil area image is Figure 5 As shown, the initial iris area image is Figure 6 As shown, the iris region image after migration is as follows Figure 7 As shown, the target iris pupil area image is as follows Figure 8 It should be noted that the grayscale value of the migrated iris region is greater than the grayscale value of the initial iris region image, so compared with the initial iris region image, the migrated iris region is darker in color, and the color difference between the migrated iris region and the pupil region image is smaller. In the embodiment of the present application, both the first weight and the second weight are not 0, so the grayscale value of the iris in the target iris pupil region image is between the grayscale value of the initial iris region image and the grayscale value of the migrated iris region.

[0125] In some embodiments, the target eye image detection model is used to perform key point recognition on the original eye image to obtain pupil key points, iris key points and eye corner key points, including:

[0126] Normalize the original eye image to obtain a standard eye image;

[0127] Mark the predicted key points on the standard eye image;

[0128] Perform image segmentation on the standard eye image according to the predicted key points to obtain the pupil key area, the iris key area and the eye corner key area;

[0129] Extract key points from the pupil key area to obtain pupil key points;

[0130] Extract key points from the key areas of the iris to obtain the key points of the iris;

[0131] The key points of the eye corner key area are extracted to obtain the eye corner key points.

[0132] The advantage of this embodiment is that the original eye image is normalized to facilitate subsequent image processing operations, and the pupil key points, iris key points and eye corner key points are obtained through image cutting and key point extraction methods, thereby realizing key point recognition and improving the detection accuracy of the key point detection method.

[0133] It should be noted that normalizing the image includes performing operations such as flipping, rotating, translating, and scaling the image to obtain a standard eye image of the same form.

[0134] It should be noted that the predicted key points are marked on the standard eye image, where the predicted key points can be the average eye key points calculated by the model through multiple sample images during the training process, and the predicted key points may deviate from the actual eye key points in the current standard eye image. In order to obtain more accurate key points, the embodiment of the present application cuts out key areas around the predicted key points by image cutting, and continuously corrects the deviation, reduces the area of ​​the key areas, and finally extracts the key points from the key areas, thereby obtaining pupil key points, iris key points and eye corner key points, which can improve the detection accuracy of the present method.

[0135] In some embodiments, the pupil key area, iris key area and eye corner key area are respectively feature extracted by the Transformer model to obtain pupil key points, iris key points and eye corner key points. In the process of feature extraction, the Transformer model can adopt the extraction method of convolutional neural network (CNN) and iterate according to a preset number of iterations (such as 3 times) to obtain the above key points. In order to avoid the gradient disappearance or gradient explosion when the convolutional neural network performs feature extraction so that accurate key points cannot be obtained, the pupil key area, iris key area and eye corner key area can be batch normalized (BN), that is, the key area is normalized to improve the stability of the model. The linear rectifier function (ReLU) can also be used as the activation function of the convolutional neural network to improve the stability of the model.

[0136] In some embodiments, the area of ​​the key area (including the pupil key area, the iris key area and the eye corner key area) cut each time is one quarter of the area of ​​the last cut. For example, if the smallest resolvable unit in the image is a pixel, the area of ​​the key area cut in the first stage is 64*64, then the area of ​​the key area cut in the second stage is 32*32.

[0137] In some embodiments, injection point detection is performed based on pupil key points, iris key points, and eye corner key points to obtain a target injection point, including:

[0138] Perform ellipse fitting on the key points of the iris to obtain the iris edge ellipse;

[0139] According to the pupil key points and the eye corner key points, the first eye line segment is obtained;

[0140] Obtain the intersection point of the first eye line segment and the iris edge ellipse as the eye reference point;

[0141] Points are selected on the first eye line segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

[0142] The advantage of this embodiment is that the target injection point is determined according to the pupil key point, the iris key point and the eye corner key point, thereby realizing the function of target injection point detection. This method can be applied to ophthalmic robots.

[0143] Specifically, the eye reference point is a point on the edge of the iris. The distance threshold value may range from 3 to 4 mm (millimeter), 3.0 to 3.5 mm, or 3.5 to 4.0 mm. For example, the distance threshold is 3.5 mm, indicating that the distance between the target injection point and the eye reference point is 3.5 mm. Other methods may also be used to set the distance threshold, such as a distance threshold of 20 pixels. In some embodiments, the target injection point is located in the corner of the eye.

[0144] In some embodiments, obtaining a first eye line segment according to the pupil key point and the eye corner key point includes:

[0145] The pupil center point is calculated based on the pupil key points;

[0146] According to the position of the eye corner key point and the preset position constraint condition, the eye corner edge point is selected from multiple eye corner key points;

[0147] Connect the pupil center point and the eye corner edge point to obtain the first eye segment.

[0148] The advantage of this embodiment is that the first eye line segment is obtained by calculation based on the pupil key point and the eye corner key point, so that the target injection point can be determined on the line segment, realizing the function of target injection point detection. This method can be applied to ophthalmic robots.

[0149] See also Fig. 9 , in some embodiments, Fig. 9 The 34 red dots in the figure represent the eye corner key points, among which the rightmost red dot (i.e., the rightmost end point of the red line segment) represents the eye corner edge point; the 8 green dots represent the iris key points, and the green ellipse represents the iris edge ellipse; the 8 white dots at the edge of the pupil area represent the pupil key points, and the white dot in the middle of the pupil key points (i.e., the leftmost end point of the red line segment) represents the pupil center point; the red line segment represents the first eye segment; the gray dot is the intersection of the first eye segment and the iris edge ellipse, representing the eye reference point; the yellow dot represents the target injection point. For example, after determining the eye reference point, you can move 20 pixels along the first eye segment toward the eye corner edge point, and use the corresponding point on the segment as the target injection point.

[0150] The embodiment of the present application further provides a controller, which includes a memory storing a computer program, and a processor implementing the above-mentioned eye injection point detection method when executing the computer program.

[0151] The specific implementation of the controller is basically the same as the specific implementation of the above-mentioned eye injection point detection method, and will not be repeated here.

[0152] The embodiment of the present application also provides a controller, the controller includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned eye injection point detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0153] See also Fig.10 , Fig.10 The hardware structure of a controller of another embodiment is illustrated, and the controller includes:

[0154] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0155] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the eye injection point detection method of the embodiment of this application;

[0156] Input / output interface 1003, used to implement information input and output;

[0157] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0158] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0159] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0160] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned eye injection point detection method is implemented.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0163] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0166] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0172] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for detecting eye injection points, characterized in that: The method comprises: Acquire a sample initial eye image from a preset image data set; the eye images in the image data set all belong to the first area; Acquire a color difference threshold; the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second area, and the first area and the second area are different areas; Performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area; Training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model; Acquire an original eye image belonging to the second area; Using the target eye image detection model, key points of the original eye image are identified to obtain pupil key points, iris key points and eye corner key points; Injection point detection is performed based on the pupil key point, the iris key point and the eye corner key point to obtain the target injection point.

2. The method for detecting eye injection points according to claim 1, characterized in that: The sample initial eye image includes a pupil area image and an initial iris area image; and performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area includes: Performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold; Performing image synthesis on the migrated iris region image and the initial iris region image to obtain a target iris pupil region image; The sample initial eye image is updated according to the target iris pupil area image to obtain the sample target eye image belonging to the second area.

3. The method for detecting eye injection points according to claim 2, characterized in that: The step of performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image includes: Performing histogram statistics based on the brightness value of the pupil area image to obtain a pupil area histogram; Performing histogram statistics based on the brightness value of the initial iris region image to obtain an iris region histogram; Performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship; The brightness value of the initial iris region image is updated according to the color difference threshold and the brightness mapping relationship to obtain the migrated iris region image.

4. The method for detecting eye injection points according to claim 2, characterized in that: The step of synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image comprises: Obtaining a weight of the initial iris region image to obtain a first weight; Obtaining the weight of the migrated iris region image to obtain a second weight, wherein the sum of the first weight and the second weight is 1; The target iris pupil area image is obtained by weighted fusion based on the first weight, the initial iris area image, the second weight and the migrated iris area image.

5. The method for detecting eye injection points according to claim 1, characterized in that: The method of performing key point recognition on the original eye image by using the target eye image detection model to obtain pupil key points, iris key points and eye corner key points includes: Normalizing the original eye image to obtain a standard eye image; Marking predicted key points on the standard eye image; Performing image segmentation on the standard eye image according to the predicted key points to obtain a pupil key area, an iris key area and an eye corner key area; Extracting key points from the pupil key area to obtain the pupil key points; Extracting key points from the iris key area to obtain the iris key points; Key points are extracted from the eye corner key area to obtain the eye corner key points.

6. The method for detecting eye injection points according to any one of claims 1 to 5, characterized in that: The step of performing injection point detection according to the pupil key point, the iris key point and the eye corner key point to obtain the target injection point includes: Performing ellipse fitting on the iris key points to obtain an iris edge ellipse; Obtaining a first eye line segment according to the pupil key point and the eye corner key point; Obtaining an intersection point between the first eye line segment and the iris edge ellipse as an eye reference point; Point selection is performed on the first eye line segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

7. The method for detecting eye injection points according to claim 6, characterized in that: The step of obtaining a first eye line segment according to the pupil key point and the eye corner key point comprises: The pupil center point is calculated according to the pupil key point; According to the position of the eye corner key point and the preset position constraint condition, selecting the eye corner edge point from the plurality of eye corner key points; The pupil center point and the eye corner edge point are connected to obtain the first eye line segment.

8. A controller, characterized in that: The controller includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. An ophthalmic robot, characterized in that: The ophthalmic robot comprises a mechanical arm, a collection module and a controller as claimed in claim 8: The controller is electrically connected to the acquisition module and the robotic arm: The acquisition module is used to acquire original eye images, and the robotic arm is used to perform injection operations according to the target injection point.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the eye injection point detection method according to any one of claims 1 to 7 is implemented.

Images