Eye injection point detection method and controller, ophthalmic robot, and storage medium

Abstract

The embodiment of the application provides a kind of eye injection point detection method and controller, ophthalmic robot and storage medium, belong to artificial intelligence technical field.The method comprises: obtaining the sample initial eye image belonging to first area from the preset image dataset;Obtain the color difference threshold;According to the color difference threshold, the sample initial eye image is color difference processed, and the sample target eye image belonging to second area is obtained;According to sample target eye image, the initial eye image detection model is trained, and the target eye image detection model is obtained;Obtain the original eye image belonging to second area;Through target eye image detection model, key point recognition is carried out to original eye image, and pupil key point, iris key point and canthus key point are obtained;According to pupil key point, iris key point and canthus key point, injection point detection is carried out, and target injection point is obtained.The embodiment of the application can improve the detection accuracy and universality of human eye key point detection method.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an eye injection point detection method and controller, an ophthalmic robot, and a storage medium. Background Art

[0002] Eye key point detection is a technology that detects facial images or videos to identify key points of the eye (such as the corners of the eyes, pupils, and irises). Eye key point detection can be applied in scenarios such as ophthalmic diagnosis (such as eye injections, ophthalmic robots). However, current eye key point detection methods are difficult to meet the detection accuracy requirements of eye injections. In addition, eye key point detection methods often target the user's eyes in a specific area, and there are differences in the eyes of different areas, so they cannot be well applied to the user's eyes in other areas outside the specific area. It can be seen that the detection accuracy of the eye key point detection method is low and its scope of application is narrow.

[0003] Therefore, how to improve the detection accuracy and universality of the human eye key point detection method has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to propose an eye injection point detection method and controller, an ophthalmic robot and a storage medium, aiming to improve the detection accuracy and universality of the human eye key point detection method, which can be applied to ophthalmic robots.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present application provides a method for detecting an eye injection point, the method comprising:

[0006] Acquire a sample initial eye image from a preset image dataset; the eye images in the image dataset all belong to the first region;

[0007] Obtaining a color difference threshold; the color difference threshold represents a difference between a pupil and an iris of an eye image belonging to a second region, the first region and the second region being different regions;

[0008] Performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area;

[0009] Training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model;

[0010] acquiring an original eye image belonging to the second region;

[0011] Performing key point recognition on the original eye image using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points;

[0012] Injection point detection is performed based on the pupil key point, the iris key point and the eye corner key point to obtain a target injection point.

[0013] In some embodiments, the sample initial eye image includes a pupil area image and an initial iris area image; and performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area includes:

[0014] performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; wherein the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold;

[0015] synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image;

[0016] The sample initial eye image is updated according to the target iris pupil area image to obtain the sample target eye image belonging to the second area.

[0017] In some embodiments, performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image includes:

[0018] Performing histogram statistics based on the brightness values ​​of the pupil area image to obtain a pupil area histogram;

[0019] Performing histogram statistics based on the brightness values ​​of the initial iris region image to obtain an iris region histogram;

[0020] Performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship;

[0021] The brightness value of the initial iris region image is updated according to the color difference threshold and the brightness mapping relationship to obtain the migrated iris region image.

[0022] In some embodiments, synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image includes:

[0023] Obtaining a weight of the initial iris region image to obtain a first weight;

[0024] Obtaining a weight of the migrated iris region image to obtain a second weight, wherein the sum of the first weight and the second weight is 1;

[0025] The target iris pupil area image is obtained by performing weighted fusion according to the first weight, the initial iris area image, the second weight, and the migrated iris area image.

[0026] In some embodiments, the step of performing key point recognition on the original eye image using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points includes:

[0027] Normalizing the original eye image to obtain a standard eye image;

[0028] Marking predicted key points on the standard eye image;

[0029] Performing image segmentation on the standard eye image according to the predicted key points to obtain a pupil key area, an iris key area, and an eye corner key area;

[0030] Extracting key points from the pupil key area to obtain the pupil key points;

[0031] Extracting key points from the iris key area to obtain the iris key points;

[0032] Key points are extracted from the eye corner key area to obtain the eye corner key points.

[0033] In some embodiments, the step of performing injection point detection based on the pupil key point, the iris key point, and the eye corner key point to obtain a target injection point includes:

[0034] Performing ellipse fitting on the iris key points to obtain an iris edge ellipse;

[0035] Obtaining a first eye line segment according to the pupil key point and the eye corner key point;

[0036] Obtaining an intersection point between the first eye line segment and the iris edge ellipse as an eye reference point;

[0037] A point is selected on the first eye line segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

[0038] In some embodiments, obtaining a first eye line segment according to the pupil key point and the eye corner key point includes:

[0039] Calculate the pupil center point according to the pupil key points;

[0040] Selecting an eye corner edge point from a plurality of eye corner key points according to the position of the eye corner key point and a preset position constraint condition;

[0041] The pupil center point and the eye corner edge point are connected to obtain the first eye line segment.

[0042] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes a controller, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect when executing the computer program.

[0043] To achieve the above-mentioned purpose, a third aspect of the embodiments of the present application provides an ophthalmic robot, comprising a robotic arm, a collection module, and the controller described in the second aspect above:

[0044] The controller is electrically connected to the acquisition module and the robotic arm:

[0045] The acquisition module is used to acquire original eye images, and the robotic arm is used to perform an injection operation according to the target injection point.

[0046] To achieve the above-mentioned purpose, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the method described in the first aspect.

[0047] The present application proposes an eye injection point detection method and controller, an ophthalmic robot, and a storage medium. The method obtains a sample initial eye image belonging to a first region from a preset image dataset; obtains a color difference threshold, which represents the difference between the pupil and iris of an eye image belonging to a second region; then performs color difference processing based on the color difference threshold and the sample initial eye image to obtain a sample target eye image belonging to the second region. Because the processed sample target eye image belongs to the second region, the target eye image detection model obtained by training a preset initial eye image detection model with the sample target eye image can accurately identify eye key points for the eyeball in the second region, preventing the model's recognition accuracy from being affected by the region to which the image belongs in the image dataset, thereby avoiding the obvious problem of only detecting eye key points for images in the first region. Furthermore, since the eyeballs of objects in different regions may differ, the color difference threshold can be adjusted based on the corresponding regions in the present application, resulting in high universality. For the original eye image belonging to the second region, the trained target eye image detection model can accurately identify the pupil key points, iris key points, and canthus key points, and then perform injection point detection, and the detected target injection point is also more accurate. This embodiment of the application can improve the detection accuracy and universality of the human eye key point detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flow chart of the eye injection point detection method provided in an embodiment of the present application;

[0049] Figure 2 is a flowchart of step 103 in Figure 1

[0050] Figure 3 is a schematic diagram of a sample initial eye image provided by an embodiment of the present application;

[0051] Figure 4 is a flowchart of step 201 in Figure 2

[0052] Figure 5 is a schematic diagram of a pupil region image provided by an embodiment of the present application;

[0053] Figure 6 is a schematic diagram of an initial iris region image provided by an embodiment of the present application;

[0054] Figure 7 is a schematic diagram of a migrated iris region image provided by an embodiment of the present application;

[0055] Figure 8 is a schematic diagram of a target iris pupil region image provided by an embodiment of the present application;

[0056] Figure 9 is a schematic diagram of a target injection point provided by an embodiment of the present application;

[0057] Figure 10 is a schematic diagram of a hardware structure of a controller provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0059] It should be noted that although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application. ​​

[0061] First, let’s analyze some of the terms used in this application:

[0062] Ocular injections, also known as intraocular drug delivery, are a treatment method that involves injecting drugs directly into the eye. They can be used to treat a variety of eye diseases. Ocular injections include intravitreal injections, subretinal injections, and suprachoroidal injections.

[0063] Ophthalmic robot: An ophthalmic robot is a surgical robot used in eye surgery.

[0064] Artificial Intelligence (AI) is a technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI encompasses image recognition, robotics, neural networks, and deep learning.

[0065] Image recognition: This refers to the technology that uses computers to process, analyze, and understand images to identify various patterns of targets and objects. Image recognition is a practical application of deep learning algorithms.

[0066] Neural Networks (NNs) are a key technology in the field of artificial intelligence. Specifically, they are mathematical or computational models that mimic the structure and function of biological neural networks. Convolutional Neural Networks (CNNs) are one of the representative algorithms for deep learning.

[0067] Rectified Linear Unit (ReLU): is a nonlinear activation function commonly used in neural networks, which can enhance the nonlinear characteristics of neural networks.

[0068] Batch Normalization (BN): refers to the normalization operation on data. It is a technology used to improve the performance and stability of neural networks and can suppress the gradient disappearance or gradient explosion of deep neural networks.

[0069] The Transformer model is a deep learning model based on the self-attention mechanism that can be used to process sequence data. The self-attention mechanism, also known as the internal attention mechanism, is an attention mechanism that associates different positions of a single sequence to calculate a representation of the same sequence.

[0070] The eye injection point detection method and controller, ophthalmic robot and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the eye injection point detection method in the embodiments of the present application is described.

[0071] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0072] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0073] It should be noted that in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain sensitive personal information of the user, the user's separate permission or consent will be obtained. After the user's separate permission or consent is clearly obtained, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0074] Figure 1 This is an optional flow chart of the eye injection point detection method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps 101 to 107.

[0075] Step 101: obtaining a sample initial eye image from a preset image dataset; the eye images in the image dataset all belong to a first region;

[0076] Step 102: Obtain a color difference threshold; the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second region, and the first region and the second region are different regions;

[0077] Step 103, performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area;

[0078] Step 104: training a preset initial eye image detection model based on the sample target eye image to obtain a target eye image detection model;

[0079] Step 105, obtaining an original eye image belonging to the second area;

[0080] Step 106: perform key point recognition on the original eye image using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points;

[0081] Step 107 : performing injection point detection based on pupil key points, iris key points, and canthus key points to obtain a target injection point.

[0082] The beneficial effects of the embodiments of the present application include but are not limited to: the sample target eye image obtained after color difference processing belongs to the second area, so the target eye image detection model obtained after training the preset initial eye image detection model with the sample target eye image can accurately identify the eye key points for the eyeball in the second area, avoiding the model's recognition accuracy being affected by the area to which the image belongs in the image dataset, thereby avoiding the model being able to only detect the eye key points for the image in the first area; the eyeballs of objects in different areas may be different, so the embodiment of the present application can adjust the color difference threshold based on the corresponding different areas, and has high universality. For the original eye image belonging to the second area, the trained target eye image detection model can accurately identify the pupil key points, iris key points and eye corner key points, and then perform injection point detection, so the detected target injection points are also more accurate. It can be seen that the embodiments of the present application can improve the detection accuracy and universality of the human eye key point detection method.

[0083] In step 101 of some embodiments, the data types in the image dataset may include images and videos. Obtaining a sample initial eye image from a preset image dataset may be to obtain a video from the image dataset, and sampling the video according to a preset sampling interval to obtain a sample initial eye image. Specifically, the video size may be about 200G, and the sampling interval may be to capture an image every two seconds. By setting a suitable sampling interval, repeated samples can be minimized, and video frames with large differences in the video can be extracted as training samples for the initial eye image detection model, thereby improving the reliability of the training model. The sample initial eye image may also be obtained by other means, not limited thereto.

[0084] It should be noted that color difference refers to the color difference between the pupil and iris in an eye image. Specifically, eye images (including the aforementioned sample initial eye images, sample target eye images, and original eye images) are all grayscale images. In this case, color difference refers to the absolute difference in grayscale values ​​between the pupil and iris regions in the eye image. The greater the color difference, the greater the brightness difference between the pupil and iris regions.

[0085] It should be noted that the color difference threshold represents the maximum difference between the pupil and the iris of the eye image belonging to the second area.

[0086] In step 102 of some embodiments, the first region and the second region are different regions, and the eyeballs of users in different regions can have large differences in iris color, color difference between the iris and the pupil, and the like. Such differences often cause the model trained using the eye images belonging to a specific region to be unable to accurately detect key points of the eye images belonging to other regions, and can even cause the trained model to be unsuitable for eye images in other regions. In the embodiments of the present application, the trained model needs to detect key points of the eye images belonging to the second region, and the eye images in the data set all belong to the first region, and re-collecting image data requires a large amount of manpower and time. In view of the above situation, the color difference threshold is obtained, and the color difference threshold represents the difference between the pupil and the iris of the eye image belonging to the second region. Then, in step 103, the sample initial eye image is color difference processed according to the color difference threshold to obtain a sample target eye image.

[0087] In step 103 of some embodiments, the sample target eye image belongs to the second region, which means that the color difference of the sample target eye image is less than the color difference threshold, that is, the sample target eye image meets the color difference characteristics of the eyeball of the second region.

[0088] In step 104 of some embodiments, a plurality of sample initial eye images can be obtained from the image data set and processed respectively to obtain a plurality of sample target eye images, and the plurality of sample target eye images are used to train the preset initial eye image detection model to obtain a target eye image detection model. Training the model by obtaining a large number of samples can improve the detection accuracy of the trained target eye image detection model in detecting eye key points of the eye images belonging to the second region, and further improve the accuracy of the target injection point obtained according to the key point detection.

[0089] In step 105 of some embodiments, the original eye image is not labeled. The original eye image can be obtained by an image acquisition device, for example, by a camera to capture the eye of a user belonging to the second region to obtain the original eye image. Specifically, a camera with a field of view (FOV) of 30° and a focal length of 10 cm can be used. In order to ensure stable light source, a circular light supplementing aperture with adjustable light can also be used when shooting, so as to ensure clear imaging of the eye when shooting.

[0090] In step 106 of some embodiments, the target eye image detection model can be a Transformer model, that is, a deep learning model based on a self-attention mechanism.

[0091] In step 107 of some embodiments, after determining the target injection point, the target injection point can provide a reference for some scenarios that require drug injection. For example, in some scenarios, a syringe can be used to inject drugs at the position of the eyeball corresponding to the target injection point. It should be noted that ocular drug injection is a treatment method that directly injects drugs into the inside of the eye. The embodiments of this application are mainly aimed at vitreous cavity injection, which means injecting drugs directly into the vitreous cavity of the eye. In addition, this method can be applied to ophthalmic robots. At present, drug injections are often performed manually, which may cause damage to the eyeball due to hand shaking during injection. Compared with traditional manual injection operations, ophthalmic robots have higher control accuracy, which can improve injection accuracy, thereby reducing damage to the eyeball caused by injection.

[0092] See also Figure 2 In some embodiments, the sample initial eye image includes a pupil area image and an initial iris area image. Step 103 may include but is not limited to steps 201 to 203:

[0093] Step 201: performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold;

[0094] Step 202: synthesize the migrated iris region image and the initial iris region image to obtain a target iris pupil region image;

[0095] Step 203 : updating the sample initial eye image according to the target iris pupil area image to obtain a sample target eye image belonging to the second area.

[0096] The advantage of this embodiment is that the color of the initial iris region image is migrated based on the color difference threshold and the pupil region image to obtain a migrated iris region image. This ensures that the color difference between the migrated iris and pupil is less than the color difference threshold, and the color difference between the iris and pupil in the eye image meets the color difference threshold constraint, making the synthesized target iris and pupil region image more closely match the color difference characteristics of the second region. The sample initial eye image is updated based on the target iris and pupil region image to obtain a sample target eye image belonging to the second region. This allows the model to be trained using samples belonging to the second region, enabling the model to accurately detect eye key points in eye images belonging to the second region, thereby improving the detection accuracy and universality of the human eye key point detection method.

[0097] Specifically, before step 201, the method may include: performing image segmentation on the sample initial eye image through a preset initial eye image detection model to obtain a pupil area image and an iris area image. In some embodiments, the eye images (including the sample initial eye images) in the image data set are all labeled, and the labels may be manually labeled or predicted by a model, which is not limited in the embodiments of the present application. The form of labels may include text annotations and video annotations. Specifically, the sample initial eye image is provided with segmentation labels and key point prediction labels. The segmentation labels are used to label the eye corner area, iris area and pupil area of ​​the eye image, and the key point prediction labels are used to label the eye corner prediction key points, iris prediction key points and pupil prediction key points of the eye image. The segmentation labels can improve the accuracy of the above-mentioned image segmentation, and the joint point prediction labels can improve the efficiency of key point detection on the sample target eye image.

[0098] See also Figure 3 In some embodiments, the sample initial eye image has segmentation labels, key point prediction labels, and viewpoint estimation labels. Figure 3 As shown in the figure, each row shows the same image and its corresponding various labels. The first column from the left is the original image, such as the grayscale image captured by a camera. The second column is the segmentation label, where the red area is the corner of the eye, the green area is the iris area, and the blue area is the pupil area. The third column is the key point prediction label, where the red point is the key point predicted for the corner of the eye, the green point is the key point predicted for the iris, and the white point is the key point predicted for the pupil. The fourth column is the viewpoint estimation label, where the green circle is the predicted eyeball range, and the line segment from the pupil to the center of the circle is used to represent the user's viewpoint.

[0099] In step 201 of some embodiments, performing color migration on the initial iris area image refers to updating the grayscale value of the initial iris area image to obtain a migrated iris area image, so as to update the color difference between the migrated iris area image and the pupil area image.

[0100] In step 202 of some embodiments, the migrated iris area image and the initial iris area image are synthesized by averaging synthesis, weighted synthesis, and other methods, which are not limited in the embodiments of the present application.

[0101] In step 203 of some embodiments, updating the sample initial eye image according to the target iris pupil area image means updating the iris area and pupil area in the sample initial eye image with the target iris pupil area image to obtain a new complete eye image, that is, the sample target eye image, which includes other information such as the corner of the eye area.

[0102] For example, the pupil area image, the initial iris area image, and the migrated iris area image are all grayscale images, and the color difference refers to the absolute value of the difference between the grayscale values ​​of the pixels in the image. The grayscale value ranges from 0 to 255, where 0 represents black and 255 represents white. For the eyeball belonging to the first area, the pupil is black and the iris is light blue; for the eyeball belonging to the second area, the pupil is black and the iris is tan. In some embodiments, red (R), green (G), and blue (B) are used to represent colors, black is represented by RGB (0, 0, 0), light blue is represented by RGB (0, 204, 255), and tan is represented by RGB (255, 102, 0). The grayscale value is calculated as (R+G+B) / 3. The grayscale value corresponding to black is 0, the grayscale value corresponding to light blue is 153, and the grayscale value corresponding to tan is 119. The sample initial eye image belonging to the first region includes a pupil region image and an initial iris region image. The grayscale value of the pupil region image is 0, and the grayscale value of the initial iris region image is 153. Therefore, the color difference of the sample initial eye image is 153-0=153. The grayscale value of the migrated iris region image is 119. Assuming that the migrated iris region image and the initial iris region image are averaged to obtain the target iris-pupil region image, the iris region grayscale value in the target iris-pupil region image is (119+153) / 2=136, and the color difference of the target iris-pupil region image is 136-0=136. Assuming a color difference threshold of 140, it can be seen that the color difference 153 of the sample initial eye image is greater than the color difference threshold, while the color difference 136 of the target iris-pupil region image is less than the color difference threshold. The color difference threshold represents the maximum difference between the pupil and iris of the eye image belonging to the second region. Therefore, the target iris-pupil region belongs to the second region. In this embodiment of the present application, the sample initial eye image is updated using the target iris and pupil region image; the resulting sample target eye image also belongs to the second region. Therefore, training the model with the sample target eye image can improve the model's accuracy in detecting eye key points in eye images belonging to the second region. Users can also adjust the color difference threshold based on the region of the eye image to be tested, thereby enhancing the universality of the detection method.

[0103] See also Figure 4 In some embodiments, step 201 may include but is not limited to steps 401 to 404:

[0104] Step 401, performing histogram statistics based on the brightness value of the pupil area image to obtain a pupil area histogram;

[0105] Step 402: performing histogram statistics based on the brightness values ​​of the initial iris region image to obtain an iris region histogram;

[0106] Step 403: performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship;

[0107] Step 404 : updating the brightness value of the initial iris region image according to the color difference threshold and the brightness mapping relationship to obtain a migrated iris region image.

[0108] The advantage of this embodiment is that the brightness values ​​of the pupil area image and the initial iris area image are represented by histograms, and the pupil area histogram and the iris area histogram are histogram matched to obtain a brightness mapping relationship;

[0109] It should be noted that, since the images are all grayscale images, the brightness value in the embodiment of the present application is equivalent to the grayscale value of the image.

[0110] It should be noted that histogram statistics refers to statistically analyzing the frequency distribution of each gray level in an image to form a grayscale histogram.

[0111] In step 403 of some embodiments, histogram matching refers to making the grayscale distribution of the initial iris region image similar to the grayscale distribution of the pupil region image based on the pupil region histogram to obtain a brightness mapping relationship.

[0112] In step 404 of some embodiments, the brightness mapping relationship can be a grayscale transformation relationship. Specifically, the brightness mapping relationship is a discontinuous mapping relationship. For example, pixels with a grayscale value of 100 in the initial iris region image are converted to pixels with a grayscale value of 140, pixels with a grayscale value of 50 are converted to pixels with a grayscale value of 80, and so on, thereby obtaining a post-migration iris region image. The brightness mapping relationship can be derived using methods such as statistical historical data, and this embodiment of the present application is not limited thereto.

[0113] In some embodiments, step 202 includes:

[0114] Obtaining the weight of the initial iris region image to obtain a first weight;

[0115] Obtaining the weight of the migrated iris region image to obtain a second weight; the sum of the first weight and the second weight is 1;

[0116] A weighted fusion is performed according to the first weight, the initial iris region image, the second weight and the migrated iris region image to obtain a target iris pupil region image.

[0117] The advantage of this embodiment is that by obtaining the weight of the initial iris area image and the weight of the migrated iris area image, the images are weightedly fused to obtain a target iris pupil area image, so that the color difference of the target iris pupil area image can be adjusted by the first weight and the second weight, so that the color difference between the pupil and the iris in the target iris pupil area image is closer to the color difference of the eye image belonging to the second area, thereby improving the flexibility of the method.

[0118] It should be noted that the iris region in the eye image includes the pupil region, and the fused target iris-pupil region image includes the iris and the pupil.

[0119] It should be noted that the first weight is used to adjust the proportion of the initial iris region image in the weighted fusion process, and the second weight is used to adjust the proportion of the migrated iris region image in the weighted fusion process.

[0120] In some embodiments, the target iris pupil area image is determined by the following formula:

[0121] S×FM1+(1-S)×FM2=FM3,

[0122] Wherein, S represents the first weight, (1-S) represents the second weight, FM1 represents the initial iris region image, FM2 represents the migrated iris region image, and FM3 represents the target iris pupil region image.

[0123] Specifically, the value range of S can be [0, 1], for example, S = 0.5. If S = 0, it means that the migrated iris area image is completely used as the target iris pupil area image.

[0124] See also Figure 5 to Figure 8 In some embodiments, the pupil area image is as follows: Figure 5 As shown, the initial iris area image is as follows Figure 6 As shown, the iris area image after migration is as follows Figure 7 As shown, the target iris pupil area image is as follows Figure 8 As shown. It should be noted that the grayscale value of the migrated iris region is greater than the grayscale value of the initial iris region image. Therefore, compared to the initial iris region image, the migrated iris region is darker, and the color difference between the migrated iris region and the pupil region image is smaller. In this embodiment of the present application, both the first weight and the second weight are non-zero. Therefore, the grayscale value of the iris in the target iris and pupil region image is between the grayscale value of the initial iris region image and the grayscale value of the migrated iris region.

[0125] In some embodiments, key point recognition is performed on the original eye image using a target eye image detection model to obtain pupil key points, iris key points, and eye corner key points, including:

[0126] Normalize the original eye image to obtain a standard eye image;

[0127] Mark the predicted key points on the standard eye image;

[0128] Perform image segmentation on the standard eye image based on the predicted key points to obtain the pupil key area, iris key area and eye corner key area;

[0129] Extract key points from pupil key areas to obtain pupil key points;

[0130] Extract key points from the key areas of the iris to obtain iris key points;

[0131] The key points of the corner of the eye are extracted to obtain the key points of the corner of the eye.

[0132] The advantage of this embodiment is that the original eye image is normalized to facilitate subsequent image processing operations, and the pupil key points, iris key points and eye corner key points are obtained through image segmentation and key point extraction methods, thereby realizing key point recognition and improving the detection accuracy of the key point detection method.

[0133] It should be noted that normalizing the image includes performing operations such as flipping, rotating, translating, and scaling the image to obtain a standard eye image of the same form.

[0134] It should be noted that the predicted key points are marked on the standard eye image, where the predicted key points can be the average eye key points calculated by the model through multiple sample images during the training process. The predicted key points may deviate from the actual eye key points in the current standard eye image. In order to obtain more accurate key points, the embodiment of the present application uses image cutting to cut out key areas around the predicted key points, continuously corrects the deviation, reduces the area of ​​the key areas, and finally extracts key points from the key areas, thereby obtaining pupil key points, iris key points, and canthus key points, which can improve the detection accuracy of the present method.

[0135] In some embodiments, the pupil key area, iris key area and eye corner key area are respectively feature extracted by the Transformer model to obtain pupil key points, iris key points and eye corner key points. In the process of feature extraction, the Transformer model can adopt the extraction method of convolutional neural network (CNN) and iterate according to a preset number of iterations (such as 3 times) to obtain the above-mentioned key points. In order to avoid the gradient disappearance or gradient explosion when the convolutional neural network performs feature extraction so that accurate key points cannot be obtained, batch normalization (BN) can be performed on the pupil key area, iris key area and eye corner key area, that is, the key area is normalized to improve the stability of the model. The linear rectifier function (ReLU) can also be used as the activation function of the convolutional neural network to improve the stability of the model.

[0136] In some embodiments, the area of ​​the key areas (including the pupil key area, iris key area, and canthus key area) cut each time is one-fourth the area of ​​the previous cut. For example, if the smallest resolvable unit in an image is a pixel, and the area of ​​the key areas cut in the first stage is 64*64, then the area of ​​the key areas cut in the second stage is 32*32.

[0137] In some embodiments, injection point detection is performed based on pupil key points, iris key points, and canthus key points to obtain a target injection point, including:

[0138] Perform ellipse fitting on the iris key points to obtain the iris edge ellipse;

[0139] According to the pupil key points and the eye corner key points, the first eye line segment is obtained;

[0140] Obtain the intersection of the first eye line segment and the iris edge ellipse as the eye reference point;

[0141] Points are selected on the first eye segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

[0142] The advantage of this embodiment is that the target injection point is determined based on the pupil key points, the iris key points and the eye corner key points, thereby realizing the function of target injection point detection. This method can be applied to ophthalmic robots.

[0143] Specifically, the eye reference point is a point on the edge of the iris. The distance threshold can range from 3 to 4 mm (millimeter), 3.0 to 3.5 mm, or 3.5 to 4.0 mm. For example, a distance threshold of 3.5 mm indicates that the distance between the target injection point and the eye reference point is 3.5 mm. Other distance thresholds can also be used, such as a distance threshold of 20 pixels. In some embodiments, the target injection point is located in the canthus area.

[0144] In some embodiments, obtaining a first eye segment according to the pupil keypoint and the eye corner keypoint includes:

[0145] The pupil center point is calculated based on the pupil key points;

[0146] According to the position of the eye corner key point and the preset position constraint conditions, the eye corner edge point is selected from multiple eye corner key points;

[0147] Connect the pupil center point and the eye corner edge point to obtain the first eye line segment.

[0148] The advantage of this embodiment is that the first eye line segment is obtained by calculation based on the pupil key point and the eye corner key point, so that the target injection point can be determined on the line segment, realizing the function of target injection point detection. This method can be applied to ophthalmic robots.

[0149] See also Figure 9 , in some embodiments, Figure 9 The 34 red dots in the image represent the eye corner keypoints, of which the rightmost red dot (i.e., the rightmost endpoint of the red line segment) represents the eye corner edge point; the eight green dots represent the iris keypoints, and the green ellipse represents the iris edge ellipse; the eight white dots at the edge of the pupil area represent the pupil keypoint, and the white dot in the middle of the pupil keypoint (i.e., the leftmost endpoint of the red line segment) represents the pupil center point; the red line segment represents the first eye segment; the gray dot is the intersection of the first eye segment and the iris edge ellipse, representing the eye reference point; and the yellow dot represents the target injection point. For example, after determining the eye reference point, you can move 20 pixels along the first eye segment toward the eye corner edge point and use the corresponding point on the line segment as the target injection point.

[0150] The present application also provides a controller including a memory storing a computer program, and a processor executing the computer program to implement the above-mentioned eye injection point detection method.

[0151] The specific implementation of the controller is basically the same as the specific embodiment of the above-mentioned eye injection point detection method, and will not be repeated here.

[0152] The present application also provides a controller comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned eye injection point detection method. The electronic device can be any smart terminal including a tablet computer, an in-vehicle computer, or the like.

[0153] See also Figure 10 , Figure 10 The hardware structure of a controller of another embodiment is shown. The controller includes:

[0154] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0155] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the eye injection point detection method of the embodiments of this application.

[0156] Input / output interface 1003, used to implement information input and output;

[0157] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0158] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0159] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0160] An embodiment of the present application further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned eye injection point detection method when executed by a processor.

[0161] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0162] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0163] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0165] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0166] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0167] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0169] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0171] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0172] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A method for detecting eye injection points, characterized in that: The method comprises: Acquire a sample initial eye image from a preset image dataset; the eye images in the image dataset all belong to the first region; Obtaining a color difference threshold; the color difference threshold represents a difference between a pupil and an iris of an eye image belonging to a second region, the first region and the second region being different regions; Performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area; Training a preset initial eye image detection model according to the sample target eye image to obtain a target eye image detection model; acquiring an original eye image belonging to the second region; Performing key point recognition on the original eye image using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points; Injection point detection is performed based on the pupil key point, the iris key point and the eye corner key point to obtain a target injection point.

2. The method for detecting eye injection points according to claim 1, wherein: The sample initial eye image includes a pupil area image and an initial iris area image; and performing color difference processing on the sample initial eye image according to the color difference threshold to obtain a sample target eye image belonging to the second area includes: performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image; wherein the color difference between the migrated iris region image and the pupil region image is less than the color difference threshold; synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image; The sample initial eye image is updated according to the target iris pupil area image to obtain the sample target eye image belonging to the second area.

3. The method for detecting eye injection points according to claim 2, wherein: The performing color migration on the initial iris region image according to the color difference threshold and the pupil region image to obtain a migrated iris region image includes: Performing histogram statistics based on the brightness values ​​of the pupil area image to obtain a pupil area histogram; Performing histogram statistics based on the brightness values ​​of the initial iris region image to obtain an iris region histogram; Performing histogram matching on the pupil area histogram and the iris area histogram to obtain a brightness mapping relationship; The brightness value of the initial iris region image is updated according to the color difference threshold and the brightness mapping relationship to obtain the migrated iris region image.

4. The eye injection point detection method according to claim 2, characterized in that: The synthesizing the migrated iris region image and the initial iris region image to obtain a target iris pupil region image includes: Obtaining a weight of the initial iris region image to obtain a first weight; Obtaining a weight of the migrated iris region image to obtain a second weight, wherein the sum of the first weight and the second weight is 1; The target iris pupil area image is obtained by performing weighted fusion according to the first weight, the initial iris area image, the second weight, and the migrated iris area image.

5. The method for detecting eye injection points according to claim 1, wherein: The method of performing key point recognition on the original eye image by using the target eye image detection model to obtain pupil key points, iris key points, and eye corner key points includes: Normalizing the original eye image to obtain a standard eye image; Marking predicted key points on the standard eye image; Performing image segmentation on the standard eye image according to the predicted key points to obtain a pupil key area, an iris key area, and an eye corner key area; Extracting key points from the pupil key area to obtain the pupil key points; Extracting key points from the iris key area to obtain the iris key points; Key points are extracted from the eye corner key area to obtain the eye corner key points.

6. The method for detecting eye injection points according to any one of claims 1 to 5, characterized in that: The step of detecting the injection point according to the pupil key point, the iris key point, and the eye corner key point to obtain the target injection point includes: Performing ellipse fitting on the iris key points to obtain an iris edge ellipse; Obtaining a first eye line segment according to the pupil key point and the eye corner key point; Obtaining an intersection point between the first eye line segment and the iris edge ellipse as an eye reference point; A point is selected on the first eye line segment according to the eye reference point and a preset distance threshold to obtain the target injection point.

7. The method for detecting eye injection points according to claim 6, wherein: The step of obtaining a first eye segment according to the pupil key point and the eye corner key point includes: Calculate the pupil center point according to the pupil key points; Selecting an eye corner edge point from a plurality of eye corner key points according to the position of the eye corner key point and a preset position constraint condition; The pupil center point and the eye corner edge point are connected to obtain the first eye line segment.

8. A controller, characterized in that: The controller includes a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. An ophthalmic robot, characterized in that: The ophthalmic robot comprises a robotic arm, a collection module, and a controller as claimed in claim 8: The controller is electrically connected to the acquisition module and the robotic arm: The acquisition module is used to acquire original eye images, and the robotic arm is used to perform an injection operation according to the target injection point.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the eye injection point detection method according to any one of claims 1 to 7 is implemented.

Images