A diopter calculation method and system based on convolutional neural network
Through the binocular diopter calculation method based on convolutional neural network, the complex problems of diopter detection equipment and algorithms in the prior art are solved, and efficient and accurate diopter detection is achieved.
Patent Information
- Application Number
- CN202410709355.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-03
- Publication Date
- 2025-02-07
- Estimated Expiration
- 2044-06-03
Smart Images
Figure CN118644446B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence, and in particular to a convolutional neural network-based diopter calculation method and system. Background Art
[0002] Myopia not only causes a continuous decline in vision and possible eye complications, but also long-term myopia will cause the eyeball to continue to stretch, thereby increasing the risk of serious eye diseases such as retinopathy and macular degeneration; especially high myopia, it is more likely to cause retinal detachment, cataracts, etc. These conditions will not only seriously affect vision, but may even lead to complete blindness. In addition, myopia will also affect a person's study, work and quality of life, and require long-term reliance on glasses or contact lenses to correct vision.
[0003] There are usually several ways to determine whether you are nearsighted: vision test, refraction test, etc. Vision test is the most basic method, which assesses the level of vision by reading the eye chart. It is simple and quick, but it cannot provide detailed information about myopia. Refraction test can measure the refractive state of the eye through instruments, and more accurately diagnose the degree of myopia and other refractive abnormalities, but it requires professional equipment and operators. Refraction test uses infrared LED light to illuminate the human eye, which will form a pattern of different light and dark on the pupil, which is also called a crescent. The size of the crescent, the ratio of black and white, and the grayscale distribution represent the human eye's myopia, hyperopia, astigmatism, axis position and other vision information.
[0004] At present, most of the detection of diopter uses traditional image processing methods. These algorithms require not only industrial-grade imaging equipment, but also high-performance central processing units. Using the principle of infrared eccentric photography optometry, a large number of binocular feature maps under different diopter states are collected, and then the diopter can be calculated through a deep learning algorithm based on a multi-layer convolutional neural network. However, unlike conventional image classification or target detection, multiple images are taken during the diopter inspection, and these images contain information related to diopter. How to process these images to obtain diopter information is an urgent problem to be solved. Summary of the invention
[0005] In order to solve the above problems, the present invention provides a binocular diopter calculation method based on a convolutional neural network, the method comprising the following steps:
[0006] Obtain eye images taken when different light groups are lit within a cycle, perform semantic segmentation on each of the eye images to obtain a pupil image, construct a pupil image sequence corresponding to the light group sequence according to the lighting order of the light groups in the light group sequence, and then obtain multiple sequences; input the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to perform feature extraction to obtain a one-dimensional vector, splice multiple one-dimensional vectors corresponding to a pupil image sequence into a feature sequence, and splice all feature sequences into a feature matrix;
[0007] Add a column of learning vectors and a row of learning vectors to the feature matrix, input the dimensionally expanded feature matrix into the ViT model, wherein an even number of encoders are connected in sequence in the ViT model, the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model;
[0008] The first row and the first column are input into two detection heads respectively, and the diopter value is further obtained through the outputs of the two detection heads.
[0009] Preferably, the method further comprises a training process, and the training process is specifically:
[0010] Input the training sample into the model to obtain the first row and the first column of the ViT model output, input the first row into the first detection head, and calculate the first loss value; input the first column into the second detection head, and calculate the second loss value; obtain the maximum value of the first loss value and the second loss value, and use the maximum value to perform back propagation to update the model parameters;
[0011] The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
[0012] Preferably, the updating of model parameters by back propagation using the maximum value is specifically as follows:
[0013] The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by back-propagation from the detection head corresponding to the maximum value.
[0014] Preferably, the performing semantic segmentation on each of the eye images to obtain the pupil image is specifically:
[0015] Binarize the eye image to obtain a region with the largest area and the largest value in the binarized region;
[0016] Obtaining a boundary in the region whose boundary curvature is within a preset range, performing circular fitting on the boundary, and obtaining a center point and radius of the circle;
[0017] Determine whether there is a bright spot in a circular area with the center point as the center and one quarter of the radius as the radius; if so, extract a pupil map from the eye image according to the center point and the radius.
[0018] Preferably, the diopter value is obtained through the output of the two detection heads, specifically:
[0019] The error of the diopter output by the two detection heads is obtained. If the error is greater than the threshold, a prompt is given to retake the image; otherwise, the average value of the output of the two detection heads is output as the diopter value.
[0020] Preferably, the step of inputting the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to extract features and obtain a one-dimensional vector is as follows:
[0021] Pre-setting three backbone networks and numbering the three backbone networks, wherein the depths of the three backbone networks increase as the numbers increase, and the sizes of the one-dimensional vectors output by the three backbone networks increase as the numbers increase;
[0022] The sequence number of the pupil image in the sequence is obtained, and the pupil image is input into a backbone network numbered as the sequence number to obtain a one-dimensional vector.
[0023] In addition, the present invention also provides a binocular refractive power calculation system based on a convolutional neural network, the system comprising the following modules:
[0024] A feature extraction module is used to obtain eye images taken when different light groups are lit within a cycle, perform semantic segmentation on each of the eye images to obtain a pupil image, and construct a pupil image sequence corresponding to the light group sequence according to the lighting order of the light groups in the light group sequence, thereby obtaining multiple sequences; according to the order of the pupil images in the sequence, the pupil images are input into the corresponding convolution-based backbone network for feature extraction to obtain a one-dimensional vector, multiple one-dimensional vectors corresponding to a pupil image sequence are spliced into a feature sequence, and all feature sequences are spliced into a feature matrix;
[0025] An encoding module is used to add a column of learning vectors and a row of learning vectors to the feature matrix, and input the dimensionally expanded feature matrix into a ViT model. An even number of encoders are connected in sequence in the ViT model, and the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model;
[0026] The decoding module is used to input the first row and the first column into two detection heads respectively and further obtain the diopter value through the outputs of the two detection heads.
[0027] Preferably, the system further comprises a training module, wherein the training module is specifically:
[0028] Input the training sample into the model to obtain the first row and the first column of the ViT model output, input the first row into the first detection head, and calculate the first loss value; input the first column into the second detection head, and calculate the second loss value; obtain the maximum value of the first loss value and the second loss value, and use the maximum value to perform back propagation to update the model parameters;
[0029] The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
[0030] Preferably, the updating of model parameters by back propagation using the maximum value is specifically as follows:
[0031] The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by back-propagation from the detection head corresponding to the maximum value.
[0032] Preferably, the performing semantic segmentation on each of the eye images to obtain the pupil image is specifically:
[0033] Binarize the eye image to obtain a region with the largest area and the largest value in the binarized region;
[0034] Obtaining a boundary in the region whose boundary curvature is within a preset range, performing circular fitting on the boundary, and obtaining a center point and radius of the circle;
[0035] Determine whether there is a bright spot in a circular area with the center point as the center and one quarter of the radius as the radius; if so, extract a pupil map from the eye image according to the center point and the radius.
[0036] Preferably, the diopter value is obtained through the output of the two detection heads, specifically:
[0037] The error of the diopter output by the two detection heads is obtained. If the error is greater than the threshold, a prompt is given to retake the image; otherwise, the average value of the output of the two detection heads is output as the diopter value.
[0038] Preferably, the step of inputting the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to extract features and obtain a one-dimensional vector is as follows:
[0039] Pre-setting three backbone networks and numbering the three backbone networks, wherein the depths of the three backbone networks increase as the numbers increase, and the sizes of the one-dimensional vectors output by the three backbone networks increase as the numbers increase;
[0040] The sequence number of the pupil image in the sequence is obtained, and the pupil image is input into a backbone network numbered as the sequence number to obtain a one-dimensional vector.
[0041] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method described above is implemented.
[0042] In order to solve the problem of multiple input images when calculating the refractive power, the present invention adopts artificial intelligence to extract features from multiple pupil images, and then adopts different backbone networks according to the influence of the lighting order in the lamp group on the pupil image. For the images taken when the lights close to the center point of the lamp group are on, a shallow backbone network is used to extract features, and for the images taken when the lights far from the center point of the lamp group are on, a deep backbone network is used to extract features. Moreover, the lengths of the features extracted by different backbone networks are different. More features are extracted when the lights far from the center are on, thereby extracting as many features as possible while ensuring the calculation speed.
[0043] In addition, when calculating attention for the extracted features, the attention is not only focused on the features of each pupil image when different light groups are on, but also by continuously transposing the feature matrix, the attention is focused on the features of the images collected when the lights are on at different distances from the center during the next attention calculation, thereby improving the effect of feature fusion. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is a flow chart of Embodiment 1;
[0045] Figure 2 This is a schematic diagram of the infrared lamp structure and lighting sequence;
[0046] Figure 3 Eye images taken for one cycle;
[0047] Figure 4 It is the pupil area extraction process;
[0048] Figure 5 is the feature matrix;
[0049] Figure 6 To increase the feature matrix of the learning vector;
[0050] Figure 7 This is a structural diagram of the second embodiment. DETAILED DESCRIPTION
[0051] In this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. In the present invention, if it involves the collection of personal privacy data, such as face, mobile phone usage information, etc., the individual's permission will be obtained in advance, including but not limited to verbal reminders, posters, mobile phone reminders, etc.; if it involves conflicts with laws and regulations, it will be produced or used within the scope permitted by laws and regulations.
[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] Embodiment 1, the present invention provides a binocular diopter calculation method based on convolutional neural network, such as Figure 1 As shown, the method comprises the following steps:
[0054] S1, obtaining eye images taken when different light groups are turned on within a cycle, performing semantic segmentation on each of the eye images to obtain a pupil image, constructing a pupil image sequence corresponding to the light group sequence according to the light-on order of the light groups in the light group sequence, and then obtaining multiple sequences; inputting the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to perform feature extraction to obtain a one-dimensional vector, splicing multiple one-dimensional vectors corresponding to a pupil image sequence into a feature sequence, and splicing all feature sequences into a feature matrix;
[0055] Figure 2The following shows the layout of the infrared lights, which include 36 infrared LED lights. Two of them are connected in series as one light group, totaling 18 groups of LED lights, and one light group is lit at a time. According to their positions, they are named [A1, A2, A3], [B1, B2, B3], [C1, C2, C3], [D1, D2, D3], [E1, E2, E3], and [F1, F2, F3]. Each light group in [] is called a light group sequence, that is, a light group sequence includes three light groups, and they are lit from the front to the back. The A1 light group is named A1 when it is lit for the first time to take pictures synchronously, and the other light groups are lit in sequence to take pictures and are named:
[0056] The first cycle forms 18 photos (1_A1, 1_A2, 1_A3, ...., 1_F1, 1_F2, 1_F3);
[0057] The second cycle is formed by analogy (2_A1, 2_A2, 2_A3,......., 2_F1, 2_F2, 2_F3);
[0058] And so on…
[0059] The fifth cycle is formed by analogy (5_A1, 5_A2, 5_A3,......., 5_F1, 5_F2, 5_F3).
[0060] Among them, 18 images taken in one cycle are as follows Figure 3 shown.
[0061] Taking a light group in a cycle as an example, after capturing an eye image when a light group is on, semantic segmentation is performed on the eye image. The purpose of semantic segmentation is to extract the image of the pupil to prevent other parts of the eye from interfering with the final result, such as eyelashes.
[0062] There are many ways to perform semantic segmentation on an image, such as the common models FCN, U-Net, etc. In order to reduce the amount of data calculated, in a more specific embodiment, the semantic segmentation of each eye image to obtain the pupil map is specifically:
[0063] Binarize the eye image to obtain a region with the largest area and the largest value in the binarized region;
[0064] First, the eye image is binarized, as Figure 4 The image in the middle is the binarized image, and then find the area with the largest area and the largest value in the binarized image. Since there is only black (pixel value 0) and white (pixel value 255) after binarization, this area is the part of the pupil illuminated by the light group.
[0065] Obtaining a boundary in the region whose boundary curvature is within a preset range, performing circular fitting on the boundary, and obtaining a center point and radius of the circle;
[0066] The pupil is approximately circular. For the convenience of calculation, a circle is fitted on the boundary area. The first step is to smooth the area to eliminate burrs. The second step is to obtain the boundary curvature of the area, add the boundary points of the boundary area within a preset range to a set, and perform circle fitting on the pixel points in the set to obtain the center point and radius of the circle.
[0067] Determine whether there is a bright spot in a circular area with the center point as the center and one quarter of the radius as the radius; if so, extract a pupil map from the eye image according to the center point and the radius.
[0068] Since the light group will also form a light spot in the pupil, especially when the second and third light groups in the light group sequence are on, and when the first light group in the light group sequence is on, the light spot is actually bright in the area. Based on this feature, it is further determined whether the center point and radius of the circle are appropriately selected. Specifically, it is determined whether there is a bright spot in the circular area with the center point as the center and one quarter of the radius as the radius. If there is, the pupil map is extracted from the eye image according to the center point and the radius.
[0069] Since the pupil is not strictly circular, in order to extract the entire pupil area, in a more specific embodiment, the pupil map is extracted from the eye image according to the center point and the radius, specifically:
[0070] The radius is enlarged by a preset multiple as a new radius, and a pupil diagram is extracted from the eye image using the new radius and a circle determined by the center point. Preferably, the multiple is 1.2 or 1.3 times. Figure 4 The rightmost image is the extracted pupil image.
[0071] Each light group sequence includes three light groups. When each light group is on, it corresponds to an eye pupil image. The eye pupil images are sorted according to the order of the lights on in the light group sequence to obtain the eye pupil image sequence of the light group sequence. There are a total of 6 light group sequences, so 6 eye pupil image sequences are obtained.
[0072] After obtaining the pupil image sequence, the backbone network is used to extract features from each pupil image in the pupil image sequence. Specifically, the pupil images are input into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to extract features and obtain a one-dimensional vector, specifically:
[0073] Pre-setting three backbone networks and numbering the three backbone networks, wherein the depths of the three backbone networks increase as the numbers increase, and the sizes of the one-dimensional vectors output by the three backbone networks increase as the numbers increase;
[0074] The sequence number of the pupil image in the sequence is obtained, and the pupil image is input into a backbone network numbered as the sequence number to obtain a one-dimensional vector.
[0075] from Figure 3 It can be seen that when the second and third light groups in each light group sequence are on, light spots will be observed in the corresponding pupil images, and the position and size of the light spots affect the final calculation results. Based on this, it is necessary to perform deep feature extraction on the second and third pupil images in the pupil image sequence. To this end, the present invention sets up three backbone networks. The first backbone network extracts features from the first pupil image in the pupil image sequence, the second backbone network extracts features from the second pupil image in the pupil image sequence, and the third backbone network extracts features from the third pupil image in the pupil image sequence. Moreover, the depth of the first backbone network is smaller than that of the second backbone network, and the depth of the second backbone network is smaller than that of the third backbone network. Since the second and third pupil image sequences include light spots, more features need to be extracted or can be extracted, and the size of the one-dimensional vectors output by the three backbone networks is also getting larger. When features are extracted such as Figure 5 As shown, each row represents the features corresponding to each pupil image sequence, and each pupil image sequence includes three sub-features, which correspond to the features of the three images in the pupil image sequence. The backbone network is used to extract features from the pupil image. The backbone network has many modes, and can adopt existing backbone networks such as VGG16, ResNet or Inception series, or can be constructed by itself, which is not specifically limited in the present invention.
[0076] Among them, multiple one-dimensional vectors corresponding to a pupil image sequence are spliced into a feature sequence by row splicing, and all feature sequences are spliced into a feature matrix by column splicing.
[0077] S2, adding a column of learning vectors and a row of learning vectors to the feature matrix, and inputting the dimensionally expanded feature matrix into the ViT model, wherein an even number of encoders are connected in sequence in the ViT model, and the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model;
[0078] After obtaining the feature matrix, add a row and a column to the feature matrix. The added row is the learning vector 1, and the added column is the learning vector 2, such as Figure 6As shown, the dimension of the feature matrix is expanded. Among them, learning vector 1 is used to use the Transformer Encoder module in the ViT model to learn the attention between the pupil images corresponding to different light group sequences. By transposing the feature matrix after dimension expansion, when proceeding to the next Transformer Encoder module, learning vector 2 is used to use the Transformer Encoder module in the ViT model to learn the attention between the pupil images when the light groups at different distances from the center are on. In a preferred embodiment, the encoder is TransformerEncoder.
[0079] S3, inputting the first row and the first column into two detection heads respectively and further obtaining the diopter value through the outputs of the two detection heads.
[0080] The first row and the first column after dimensional expansion, that is, learning vector 1 and learning vector 2, continuously learn the features of the input pupil image during the training process, and then learning vector 1 and learning vector 2 are respectively input into two independent detection heads, and then the refractive power value is obtained through the output of the two detection heads.
[0081] In a more specific embodiment, the diopter value is obtained through the output of the two detection heads, specifically:
[0082] The error of the diopter output by the two detection heads is obtained. If the error is greater than the threshold, a prompt is given to retake the image; otherwise, the average value of the output of the two detection heads is output as the diopter value.
[0083] In another embodiment, the method further comprises step S4, obtaining the diopter of the pupil image taken in each of the five cycles, removing the maximum and minimum values, calculating the average of the remaining diopters, and taking the average as the final diopter value. One cycle means that all infrared light groups are turned on once.
[0084] In another embodiment, the method further includes a training sample construction process, specifically, taking all pupil images taken in one cycle as a sample, and labeling the sample according to the refractive index of all pupil images taken in this cycle, and the labeled value is the refractive index value.
[0085] During the training process, the training sample is input into the model to obtain the first row and the first column of the ViT model output, the first row is input into the first detection head, and the first loss value is calculated; the first column is input into the second detection head, and the second loss value is calculated; the maximum value of the first loss value and the second loss value is obtained, and the maximum value is used for back propagation to update the model parameters;
[0086] The number of training samples directly affects the accuracy of the final prediction. The present invention trains the two detection heads separately. Specifically, the learning vector 1 is input into the first detection head to calculate the first loss value, and the learning vector 2 is input into the second detection head to calculate the second loss value. The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by backpropagation from the detection head corresponding to the maximum value.
[0087] The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
[0088] The model parameters will be updated once for each batch, that is, back propagation will be performed once. The present invention sets two detection heads, and the same training samples can be repeatedly used for training, and the model can be continuously updated to improve the utilization of the samples. Of course, one detection head can also be used, that is, in a replaceable embodiment, step S3 is to splice the first row and the first column and input them into a detection head to obtain the value of the diopter. The present invention does not specifically limit the structure of the detection head, for example, MLP is used. Table 1 shows a comparison of the effects of the model using one detection head and two detection heads under the same preset training set.
[0089] Table 1
[0090]
[0091] In addition, the present invention also provides a binocular diopter calculation system based on convolutional neural network, such as Figure 7 As shown, the system includes the following modules:
[0092] A feature extraction module is used to obtain eye images taken when different light groups are lit within a cycle, perform semantic segmentation on each of the eye images to obtain a pupil image, and construct a pupil image sequence corresponding to the light group sequence according to the lighting order of the light groups in the light group sequence, thereby obtaining multiple sequences; according to the order of the pupil images in the sequence, the pupil images are input into the corresponding convolution-based backbone network for feature extraction to obtain a one-dimensional vector, multiple one-dimensional vectors corresponding to a pupil image sequence are spliced into a feature sequence, and all feature sequences are spliced into a feature matrix;
[0093] An encoding module is used to add a column of learning vectors and a row of learning vectors to the feature matrix, and input the dimensionally expanded feature matrix into a ViT model. An even number of encoders are connected in sequence in the ViT model, and the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model;
[0094] The decoding module is used to input the first row and the first column into two detection heads respectively and further obtain the diopter value through the outputs of the two detection heads.
[0095] Preferably, the system further comprises a training module, wherein the training module is specifically:
[0096] Input the training sample into the model to obtain the first row and the first column of the ViT model output, input the first row into the first detection head, and calculate the first loss value; input the first column into the second detection head, and calculate the second loss value; obtain the maximum value of the first loss value and the second loss value, and use the maximum value to perform back propagation to update the model parameters;
[0097] The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
[0098] Preferably, the updating of model parameters by back propagation using the maximum value is specifically as follows:
[0099] The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by back-propagation from the detection head corresponding to the maximum value.
[0100] Preferably, the performing semantic segmentation on each of the eye images to obtain the pupil image is specifically:
[0101] Binarize the eye image to obtain a region with the largest area and the largest value in the binarized region;
[0102] Obtaining a boundary in the region whose boundary curvature is within a preset range, performing circular fitting on the boundary, and obtaining a center point and radius of the circle;
[0103] Determine whether there is a bright spot in a circular area with the center point as the center and one quarter of the radius as the radius; if so, extract a pupil map from the eye image according to the center point and the radius.
[0104] Preferably, the diopter value is obtained through the output of the two detection heads, specifically:
[0105] The error of the diopter output by the two detection heads is obtained. If the error is greater than the threshold, a prompt is given to retake the image; otherwise, the average value of the output of the two detection heads is output as the diopter value.
[0106] Preferably, the step of inputting the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to extract features and obtain a one-dimensional vector is as follows:
[0107] Pre-setting three backbone networks and numbering the three backbone networks, wherein the depths of the three backbone networks increase as the numbers increase, and the sizes of the one-dimensional vectors output by the three backbone networks increase as the numbers increase;
[0108] The sequence number of the pupil image in the sequence is obtained, and the pupil image is input into a backbone network numbered as the sequence number to obtain a one-dimensional vector.
[0109] Embodiment 3, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in embodiment 1 is implemented.
[0110] Through the description of the above implementation methods, technicians in this field can clearly understand that each implementation method can be implemented by adding a necessary general hardware platform, and of course can also be implemented by combining hardware and software. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a computer product, and the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it, and other embodiments may also be used. Although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A binocular refractive power calculation method based on convolutional neural network, characterized in that: The method comprises the following steps: Obtain eye images taken when different light groups are lit within a cycle, perform semantic segmentation on each of the eye images to obtain a pupil image, construct a pupil image sequence corresponding to the light group sequence according to the lighting order of the light groups in the light group sequence, and then obtain multiple sequences; input the pupil images into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence to perform feature extraction to obtain a one-dimensional vector, splice multiple one-dimensional vectors corresponding to a pupil image sequence into a feature sequence, and splice all feature sequences into a feature matrix; Add a column of learning vectors and a row of learning vectors to the feature matrix, input the dimensionally expanded feature matrix into the ViT model, wherein an even number of encoders are connected in sequence in the ViT model, the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model; Inputting the first row and the first column into two detection heads respectively and further obtaining the diopter value through the outputs of the two detection heads; The pupil images are input into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence for feature extraction to obtain a one-dimensional vector, specifically: three backbone networks are pre-set and numbered, wherein the depths of the three backbone networks increase with the increase of the numbers, and the sizes of the one-dimensional vectors output by the three backbone networks increase with the increase of the numbers; the sequence number of the pupil image in the sequence is obtained, and the pupil image is input into the backbone network numbered with the sequence number to obtain a one-dimensional vector; the first backbone network extracts features from the first pupil image in the pupil image sequence, the second backbone network extracts features from the second pupil image in the pupil image sequence, and the third backbone network extracts features from the third pupil image in the pupil image sequence, and the depth of the first backbone network is less than that of the second backbone network, and the depth of the second backbone network is less than that of the third backbone network; the sequence number of the pupil image in the sequence is sorted as follows: in each light group sequence, the light groups are sorted according to their distance from the center point of all light group sequences, and the sequence numbers increase gradually from near to far; The method of adding a column of learning vectors and a row of learning vectors to the feature matrix and inputting the feature matrix after dimension expansion into the ViT model is as follows: the added row is learning vector 1, and the added column is learning vector 2, wherein learning vector 1 is used to use the Transformer Encoder module in the ViT model to learn the attention between pupil images corresponding to different light group sequences, and by transposing the feature matrix after dimension expansion, when proceeding to the next Transformer Encoder module, learning vector 2 is used to use the Transformer Encoder module in the ViT model to learn the attention between pupil images when light groups at different distances from the center are on.
2. The method according to claim 1, characterized in that The method further includes a training process, which specifically includes: Input the training sample into the model to obtain the first row and the first column of the ViT model output, input the first row into the first detection head, and calculate the first loss value; input the first column into the second detection head, and calculate the second loss value; obtain the maximum value of the first loss value and the second loss value, and use the maximum value to perform back propagation to update the model parameters; The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
3. The method according to claim 2, characterized in that The use of the maximum value to perform back propagation to update the model parameters is specifically as follows: The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by back-propagation from the detection head corresponding to the maximum value.
4. The method according to claim 1, characterized in that The semantic segmentation of each eye image to obtain the pupil image is specifically: Binarize the eye image to obtain a region with the largest area and the largest value in the binarized region; Obtaining a boundary in the region whose boundary curvature is within a preset range, performing circular fitting on the boundary, and obtaining a center point and radius of the circle; Determine whether there is a bright spot in a circular area with the center point as the center and one quarter of the radius as the radius; if so, extract a pupil map from the eye image according to the center point and the radius.
5. The method according to claim 1, characterized in that The diopter value is obtained through the output of the two detection heads, specifically: The error of the diopter output by the two detection heads is obtained. If the error is greater than the threshold, a prompt is given to retake the image; otherwise, the average value of the output of the two detection heads is output as the diopter value.
6. A binocular refractive power calculation system based on convolutional neural network, characterized in that: The system includes the following modules: A feature extraction module is used to obtain eye images taken when different light groups are lit within a cycle, perform semantic segmentation on each of the eye images to obtain a pupil image, and construct a pupil image sequence corresponding to the light group sequence according to the lighting order of the light groups in the light group sequence, thereby obtaining multiple sequences; according to the order of the pupil images in the sequence, the pupil images are input into the corresponding convolution-based backbone network for feature extraction to obtain a one-dimensional vector, multiple one-dimensional vectors corresponding to a pupil image sequence are spliced into a feature sequence, and all feature sequences are spliced into a feature matrix; The pupil images are input into the corresponding convolution-based backbone network according to the order of the pupil images in the sequence for feature extraction to obtain a one-dimensional vector, specifically: three backbone networks are pre-set and numbered, wherein the depths of the three backbone networks increase with the increase of the numbers, and the sizes of the one-dimensional vectors output by the three backbone networks increase with the increase of the numbers; the sequence number of the pupil images in the sequence is obtained, and the pupil images are input into the backbone network numbered with the sequence number to obtain a one-dimensional vector; the first backbone network extracts features from the first pupil image in the pupil image sequence, the second backbone network extracts features from the second pupil image in the pupil image sequence, and the third backbone network extracts features from the third pupil image in the pupil image sequence, and the depth of the first backbone network is less than that of the second backbone network, and the depth of the second backbone network is less than that of the third backbone network; in each light group sequence, the light groups are sorted according to their distance from the center point of all light group sequences, and the sequence numbers increase gradually from near to far; An encoding module is used to add a column of learning vectors and a row of learning vectors to the feature matrix, and input the dimensionally expanded feature matrix into a ViT model. An even number of encoders are connected in sequence in the ViT model, and the output of each encoder is transposed, and the transposed vector is used as the input of the next encoder to obtain the first row and the first column of the feature matrix output by the ViT model; The step of adding a column of learning vectors and a row of learning vectors to the feature matrix and inputting the feature matrix after dimension expansion into the ViT model is as follows: the added row is learning vector 1, and the added column is learning vector 2, wherein learning vector 1 is used to learn the attention between pupil images corresponding to different light group sequences by using the Transformer Encoder module in the ViT model, and by transposing the feature matrix after dimension expansion, when proceeding to the next Transformer Encoder module, learning vector 2 is used to learn the attention between pupil images when light groups at different distances from the center are on by using the Transformer Encoder module in the ViT model; The decoding module is used to input the first row and the first column into two detection heads respectively and further obtain the diopter value through the outputs of the two detection heads.
7. The system according to claim 6, characterized in that The system also includes a training module, which is specifically: Input the training sample into the model to obtain the first row and the first column of the ViT model output, input the first row into the first detection head, and calculate the first loss value; input the first column into the second detection head, and calculate the second loss value; obtain the maximum value of the first loss value and the second loss value, and use the maximum value to perform back propagation to update the model parameters; The above operation is performed multiple times for each batch of training samples until the change between the first loss value and the second loss value is less than a preset value.
8. The system according to claim 7, characterized in that The use of the maximum value to perform back propagation to update the model parameters is specifically as follows: The parameters of the detection heads other than the maximum value are kept unchanged, and the model parameters are updated by back-propagation from the detection head corresponding to the maximum value.
9. A computer storage device, wherein a computer program is stored on the storage device, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.
Citation Information
Patent Citations
Vision detection algorithm based on portable infrared camera
CN110279391A
KR20200077085A