Automatic optometry auxiliary control method and device and medium

Abstract

The invention provides an automatic optometry auxiliary control method and device and a medium, and belongs to the technical field of optometry units. The method comprises the following steps: after an optometry preparation signal from an optometry inductive sensor is received, acquiring a first image from image acquisition equipment in real time; and inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry object. On the basis of the facial feature information and a preset eyeball region extraction rule, eyeball region position information corresponding to the optometry object is determined, a motor control instruction is generated according to the eyeball region position information, and a motor drives an optometry lens assembly to move to an eyeball region matching position according to the motor control instruction. And inputting the second image into a pre-trained eye feature extraction model to determine eye feature information of the optometry object. And based on a preset random forest model and the eye feature information, determining a pupil focusing result corresponding to the optometry object, and sending the pupil focusing result to a downstream optometry task.

Description

Technical Field

[0001] The present application relates to the technical field of ophthalmometers, and in particular to an automatic ophthalmometer auxiliary control method, device and medium. Background Art

[0002] In the field of ophthalmology and vision correction, computer ophthalmometer is an important diagnostic tool. Its accuracy and convenience are directly related to the accurate assessment of the patient's refractive state and the formulation of personalized vision correction plan. However, traditional computer ophthalmometer has some significant shortcomings in use.

[0003] First, traditional computer ophthalmometers are highly dependent on the manual operation of optometrists when performing optometry. Optometrists move the joystick up, down, left, and right to accurately adjust the corneal reflection point cursor to the center of the pupil. This process is not only cumbersome, but also has high requirements on the technical proficiency and professional ability of optometrists. This means that in order to obtain accurate optometry results, medical institutions need to invest a lot of talent costs to train and hire experienced optometrists.

[0004] Secondly, due to the high degree of manualization of the optometry process, the examination results often carry the subjective bias of the operator. Different optometrists may come to different conclusions about the refractive status of the same patient due to their personal experience, technical level or judgment criteria. This subjectivity not only reduces the stability of the optometry results, but may also affect the effectiveness and safety of subsequent vision correction plans.

[0005] In addition, the optometry process of traditional computer ophthalmometers is relatively time-consuming and inefficient. With the increasing number of patients, medical institutions are facing tremendous pressure for diagnosis and treatment. Traditional optometry methods not only cannot meet the growing demand for diagnosis and treatment, but may also reduce patients' medical experience and satisfaction due to long waiting times.

[0006] Based on this, there is an urgent need for a technical solution that can automatically assist in completing pupil focus-related controls during the optometry process efficiently, accurately and at low cost. Summary of the invention

[0007] The embodiments of the present application provide an automatic optometry auxiliary control method, device and medium, which are used to solve the technical problems that the pupil focus in the current optometry process is overly dependent on manual operation, the optometry process is not efficient enough and the labor cost is high, which affects the user's medical experience.

[0008] On the one hand, an embodiment of the present application provides an automatic optometry assisted control method, the method comprising:

[0009] After receiving the optometry preparation signal from the optometry sensing sensor, a first image is acquired from the image acquisition device in real time; wherein the first image at least includes the face of the optometry subject;

[0010] Inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject;

[0011] Based on the facial feature information and the preset eyeball region extraction rule, determine the eyeball region position information corresponding to the optometry object, generate a motor control instruction according to the eyeball region position information, so that the motor drives the optometry lens assembly to move to the eyeball region matching position according to the motor control instruction;

[0012] Inputting the second image into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject; wherein the second image is obtained by image cropping based on the eyeball area position information;

[0013] Based on the preset random forest model and the eye feature information, the pupil focus result corresponding to the optometry object is determined, so as to send the pupil focus result to the downstream optometry task.

[0014] In one implementation of the present application, before inputting the first image into a pre-trained facial feature recognition model to determine the facial feature information of the optometry subject, the method further includes:

[0015] Acquire a number of image training samples and perform normalization processing; wherein the image training samples are pre-labeled with distortion labels and include corresponding distortion-free reference images; the distortion labels correspond to a set of distortion comparison parameters of the distorted images in the image training samples and the corresponding distortion-free reference images; the image training samples are acquired by a device of the same type as the image acquisition device; the image acquisition device is a wide-angle camera;

[0016] Input the normalized image training samples into the first recognition sub-model to be trained until the test evaluation index corresponding to the first recognition sub-model to be trained meets the preset first training end condition, so as to obtain the trained first recognition sub-model to construct the facial feature recognition model with the second recognition sub-model; the first recognition sub-model includes an encoder and a decoder.

[0017] In one implementation of the present application, inputting the first image into a pre-trained facial feature recognition model to determine the facial feature information of the optometry subject specifically includes:

[0018] Inputting the third image after the distortion correction processing by the first recognition sub-model into the second recognition sub-model;

[0019] By using the second recognition sub-model, the preset sliding window is moved in the normalized third image according to a preset order, and the pixel features in the movement coverage area are extracted by the convolution layer each time to determine whether there is facial pixel information in the movement coverage area; the facial pixel information is obtained based on the edge information and texture information corresponding to the extracted features;

[0020] A facial bounding box is determined based on each facial pixel information, so as to determine the facial feature information of the optometry subject according to the position information of the facial bounding box in the third image.

[0021] In one implementation of the present application, based on the facial feature information and the preset eye area extraction rule, determining the eye area position information corresponding to the optometry object specifically includes:

[0022] Determine feature information of facial features based on the facial feature information; wherein the feature information of facial features at least includes position information of the following parts: eyeballs, nose, mouth, ears, and eyebrows;

[0023] Filtering any multiple parts in the facial feature region feature information and generating multiple part sets; each part set includes at least one part that is different from other part sets;

[0024] Based on cluster analysis, a part association list is obtained by processing a number of preset standard facial feature image samples, and the relative position information of the eyeball area corresponding to each of the part sets is matched; wherein the part association list includes part position vectors under different part combinations and corresponding part combination credibility; the relative position information of the eyeball area is the part position vector of the eyeball area relative to other parts in a pre-constructed facial coordinate system;

[0025] Determine the eyeball region cluster position vector according to the part combination credibility of each part set in the part association list and the weighted vector sum corresponding to the part position vector, and extract the eyeball region cluster position information according to the eyeball region cluster position vector and a preset position comparison table;

[0026] The eyeball region clustering position information is compared with the facial features region feature information to determine whether the facial feature recognition model needs to be updated based on the comparison result.

[0027] In one implementation of the present application, the second image is input into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject, specifically including:

[0028] Performing cropping processing on the image according to the eye area position information to obtain the second image;

[0029] Inputting the second image into the pre-trained eye feature extraction model to determine a preset number of target feature points in the second image; the preset number of target feature points constitute the eye contour information of the optometry subject; the eye contour information at least includes multiple same-color area boundaries of the eyeball;

[0030] Determining the feature point coordinates of each of the target feature points based on a preset plane rectangular coordinate system established with the eye corner as the origin;

[0031] The coordinates of each of the feature points are used as the eye feature information.

[0032] In one implementation of the present application, before determining the pupil focus result corresponding to the optometry object based on the preset random forest model and the eye feature information, and sending the pupil focus result to the downstream optometry task, the method further includes:

[0033] Inputting a number of eye image samples into the pre-trained eye feature extraction model to determine feature information of each eye sample;

[0034] Based on the user's annotation operation, determine the pupil center point information corresponding to each of the eye images;

[0035] The feature information of each eye sample is used as a regression feature, and the pupil center point information corresponding to each eye sample feature information is used as a regression target. Regression prediction is performed through the random forest model to be trained, and the mean square error and goodness of fit corresponding to the regression prediction are determined, so as to perform parameter tuning on the random forest model to be trained based on the mean square error and the goodness of fit until the mean square error and the goodness of fit meet the preset second training end condition, so as to obtain the preset random forest model that has completed the training.

[0036] In one implementation of the present application, the distortion control parameter group includes at least one or more of the following: radial distortion parameter, tangential distortion parameter, distortion center, focal length, and principal point offset.

[0037] In one implementation of the present application, before receiving the optometry preparation signal from the optometry sensing sensor, the method further includes:

[0038] Acquire an initial optometry signal from the optometry sensing sensor; wherein the optometry sensing sensor is a thermistor; the initial optometry signal is generated when the forehead of the optometry subject contacts the thermistor;

[0039] When the duration corresponding to the initial optometry signal is longer than the preset duration, the corresponding initial optometry signal is used as the optometry preparation signal;

[0040] Otherwise, the initial optometry signal and the accumulated duration are discarded.

[0041] On the other hand, an embodiment of the present application further provides an automatic optometry auxiliary control device, the device comprising:

[0042] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform an automatic optometry assisted control method as described above.

[0043] On the other hand, an embodiment of the present application further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions can execute an automatic optometry assisted control method as described above.

[0044] Compared with the prior art, the present invention has the following significant effects:

[0045] Through the above technical scheme, the present application uses a facial feature recognition model to extract facial feature information and extract the eye area position information to achieve coarse focusing of the eye area. Subsequently, the eye feature extraction model is used to extract eye feature information, and the random forest algorithm is used for regression fitting to obtain the pupil focusing result, so as to achieve accurate and efficient focusing of the pupil. In this way, the eye area can be quickly located without excessive reliance on manual labor, which reduces the subjective component in the optometry process, saves a lot of manpower and material resources, and can improve the user's medical experience. The above scheme can overcome the limitations of traditional computer optometry instruments, improve the accuracy and stability of optometry results, while reducing the talent cost and time cost of medical institutions, and providing patients with better quality and more efficient ophthalmic diagnosis and treatment services. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 A schematic diagram of a flow chart of an automatic optometry auxiliary control method in an embodiment of the present application;

[0048] Figure 2 Another schematic diagram of a flow chart of an automatic optometry auxiliary control method in an embodiment of the present application;

[0049] Figure 3 This is a schematic diagram of the structure of an automatic optometry auxiliary control device in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0051] The computer ophthalmometers that are widely used at present have high requirements on the operator's technical proficiency and professional ability, and their inspection results also carry the operator's subjective bias.

[0052] In addition, the pupil localization algorithms used by traditional ophthalmometers, such as the pupil localization method based on Hough transform circle detection and the pupil localization algorithm based on regional projection, mostly use the threshold method to binarize the eye image and realize the pupil localization through the pupil connected domain screening algorithm. However, the traditional pupil connected domain screening condition has weak constraints and low positioning accuracy. It is difficult to effectively locate the pupil in complex eye images, which affects the overall efficiency of the optometry process.

[0053] Based on this, the embodiments of the present application provide an automatic optometry assisted control method, device and medium to solve the technical problems that the pupil focus in the current optometry process is overly dependent on manual operation, the optometry process is not efficient enough and the labor cost is high, which affects the user's medical experience.

[0054] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.

[0055] The present application embodiment provides an automatic optometry auxiliary control method, which is applied to an optometry instrument, such as Figure 1 As shown, the method may include steps S101-S105:

[0056] S101, after receiving an optometry preparation signal from an optometry sensing sensor, the microprocessor acquires a first image from an image acquisition device in real time.

[0057] The first image at least includes the face of the optometry subject.

[0058] First, in the embodiment of the present application, before receiving the optometry preparation signal from the optometry sensing sensor, the microprocessor further includes:

[0059] Obtain an initial optometry signal from an optometry sensor. The optometry sensor is a thermistor. The initial optometry signal is generated when the forehead of the optometry subject contacts the thermistor. When the duration corresponding to the initial optometry signal is greater than the preset duration, the corresponding initial optometry signal is used as an optometry preparation signal. Otherwise, the initial optometry signal and the accumulated duration are discarded.

[0060] In other words, the present application uses a thermistor as an optometry sensor, which can be set in the user's forehead contact area of ​​the optometry instrument, and the user is the optometry object. After the user puts his forehead against the user's forehead contact area, the thermistor will change its resistance value due to heat, and generate the above-mentioned initial optometry signal. And by setting a duration as a limiting condition, a real-time optometry preparation signal is obtained, while avoiding the execution of the optometry process due to accidental touch.

[0061] Specifically, the present application selects a highly sensitive and fast-responding thermistor as an optometry sensing sensor, designs a low-noise and high-precision signal processing circuit, and converts the resistance change of the thermistor into an easily measurable voltage or current signal. The analog-to-digital converter (ADC) module built into the microprocessor converts the conditioned analog signal into a digital signal. A optometry monitoring threshold is preset in the microprocessor. The optometry monitoring threshold is set according to the calibration data of the thermistor and the system requirements, and is used to determine the forehead contact and trigger the optometry process. When the change in the digital signal converted by the ADC module reaches or exceeds the optometry monitoring threshold, it is considered that the forehead has effectively contacted the thermistor, and an initial optometry signal will be generated at this time. In order to prevent misjudgment caused by short-term contact or false touch, a de-jitter algorithm is implemented, that is, after detecting the first triggering of the optometry monitoring threshold, wait for a preset time length such as 500 milliseconds. If the signal continues to remain above the threshold during this period, the trigger is confirmed to be valid, otherwise it is considered to be a false touch and the state is reset.

[0062] The above steps are the preparation stage of the optometry process. Figure 2 As shown, the forehead is placed close to the thermistor (ie, thermistor), the thermistor changes, the main control system receives the signal, and the test starts. The main control system is a system with the above-mentioned microprocessor.

[0063] It should be noted that the microprocessor as the execution subject of the automatic optometry auxiliary control method is only for example, and the execution subject is not limited to the microprocessor, and this application does not make specific restrictions on this. The microprocessor can be set in the optometry instrument.

[0064] The image acquisition device may be a camera placed in front of the optometry subject, capable of acquiring a photograph of the optometry subject's face.

[0065] S102, the microprocessor inputs the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject.

[0066] In the embodiment of the present application, before inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject, the method further includes:

[0067] Obtain several image training samples and perform normalization processing. The image training samples are pre-labeled with distortion labels and include corresponding distortion-free reference images. The distortion labels correspond to the distortion comparison parameter group of the distorted images in the image training samples and their corresponding distortion-free reference images. The image training samples are acquired by a device of the same type as the image acquisition device. The image acquisition device is a wide-angle camera with a resolution of, for example, 4K. Input each image training sample after normalization processing into the first recognition sub-model to be trained until the test evaluation index corresponding to the first recognition sub-model to be trained meets the preset first training end condition, and obtain the trained first recognition sub-model to construct a facial feature recognition model with the second recognition sub-model. The first recognition sub-model includes an encoder and a decoder.

[0068] That is to say, the present application trains the first recognition sub-model in the facial feature recognition model through image training samples with distortion labels. The first recognition sub-model can be a convolutional neural network model (CNN), which is used to extract deep features of the image and reconstruct image details to achieve image correction. Among them, the distortion label corresponds to a distortion control parameter group, including at least one or more of the following: radial distortion parameters, tangential distortion parameters, distortion center, focal length, principal point offset, and the image training sample includes a distorted image and its distortion-free parameter image, and both images carry distortion control parameters.

[0069] In order to obtain the facial image of the optometry object more comprehensively and in a limited space, it is necessary for this application to use a wide-angle camera as an image acquisition device to acquire the above-mentioned first image. However, due to the special optical constraint principle of the ultra-wide-angle camera, the wide-angle image presents a nonlinear image distortion that becomes more serious from the center of the image to the edge. This makes it difficult for the captured image to directly use the existing vision and image processing operators to perform related tasks. The solution is to correct the wide-angle image into a perspective projection image that is easy for the computer to understand through image processing. Therefore, this application proposes the above-mentioned first recognition sub-model, and performs distortion correction of the first image through the first recognition sub-model. Among them, the above-mentioned test evaluation indicators can be the model output accuracy, or the loss function value, or the accuracy, recall rate, F1 score, etc., which can be set by the user during actual use, and this application does not make specific restrictions on this.

[0070] This application uses a convolutional neural network model to estimate the radial distortion parameters, tangential distortion parameters, distortion center, focal length, principal point offset, etc. of a wide-angle image. By learning the intermediate form of the pixel position change from input to output of the photo, it clearly indicates how the pixel at each known position in the distorted image is mapped to a new position, so that all pixels in the input image can learn the distortion flow prediction directly related to them, thereby obtaining a more accurate corrected image. Among them, the first recognition sub-model encoder is responsible for extracting deep features of the input image, especially features related to geometric structures and non-natural patterns. It gradually reduces the spatial dimension of the feature map through multi-layer convolution and pooling (or downsampling) operations, while increasing the number of channels of the feature map to capture richer information. The decoder restores the low-resolution, high-dimensional feature map output by the encoder to the original image size or target size, while reconstructing image details, especially details of non-natural structures.

[0071] In addition, when training the first recognition sub-model, it is also necessary to set an appropriate learning rate, batch size, number of iterations, etc., which are set by the developer according to the actual usage scenario before training, and this application does not make specific restrictions on this. The data set is divided into a training set, a validation set, and a test set. Use the training set data to train the first recognition sub-model, monitor the performance on the validation set, adjust the model parameters and training strategy in a timely manner, and apply the early stopping method to prevent overfitting. Use the test set to evaluate the performance of the model, including indicators such as accuracy, recall rate, and F1 score. At the same time, you can also visualize the image output by the decoder to check the effect of distortion correction.

[0072] In the embodiment of the present application, the first image is input into a pre-trained facial feature recognition model to determine the facial feature information of the optometry object, specifically including:

[0073] The third image after the distortion correction processing by the first recognition sub-model is input into the second recognition sub-model. Through the second recognition sub-model, the preset sliding window is moved in the third image after normalization according to a preset order, and the pixel features in the moving coverage area are extracted through the convolution layer each time to determine whether there is facial pixel information in the moving coverage area. The facial pixel information is obtained based on the edge information and texture information corresponding to the extracted features. Based on the information of each facial pixel, the facial bounding box is determined to determine the facial feature information of the optometry subject according to the position information of the facial bounding box in the third image.

[0074] That is, the present application corrects the distortion of the first image through the trained first recognition sub-model to obtain the third image, and then normalizes the third image to adjust the size and color attributes of the third image to the standard format used in the training of the second recognition sub-model.

[0075] Next, the facial feature information in the third image is extracted through the pre-trained second recognition sub-model. Among them, the second recognition sub-model can also use a CNN model. The second recognition sub-model uses a sliding window technology to slide a preset sliding window of a preset size on the third image. The preset size can be pre-set by the developer according to the actual usage scenario, and this application does not make specific restrictions on this. The second recognition sub-model will detect in real time whether there is a face in the preset sliding window. Specifically, the convolution layer of CNN can be used to extract features of the image in the window, and extract features such as the edge and texture of the image. Based on the extracted features, the bounding box of the face (facial bounding box), that is, the position and size of the face in the image, is determined, and then the facial feature information is obtained. Facial feature information can be understood as a high-dimensional feature vector containing key facial information such as facial features.

[0076] The above steps can be understood as Figure 2 The image processing steps in the embodiment include taking photos with a wide-angle camera, processing photo distortion, and identifying facial feature points using a face recognition algorithm (a second recognition sub-model).

[0077] S103, the microprocessor determines the eye area position information corresponding to the eye test object based on the facial feature information and the preset eye area extraction rules, and generates a motor control instruction according to the eye area position information, so that the motor drives the eye test lens assembly to move to the eye area matching position according to the motor control instruction.

[0078] In the embodiment of the present application, based on the facial feature information and the preset eyeball area extraction rules, the eyeball area position information corresponding to the optometry object is determined, specifically including:

[0079] According to the facial feature information, the feature information of the facial features region is determined. The feature information of the facial features region includes at least the position information of the following parts: eyeball, nose, mouth, ear, and eyebrow. Any multiple parts in the feature information of the facial features region are selected, and multiple part sets are generated. Each part set includes at least one part that is different from other part sets. Based on the part association list obtained by cluster analysis and processing of a number of preset standard facial features image samples, the relative position information of the eyeball region corresponding to each part set is matched. The part association list includes the part position vectors under different part combinations and the corresponding part combination credibility. The relative position information of the eyeball region is the part position vector of the eyeball region relative to other parts in the pre-constructed facial coordinate system. According to the part combination credibility of each part set in the part association list and the weighted vector sum corresponding to the part position vector, the eyeball region cluster position vector is determined, and the eyeball region cluster position information is extracted according to the eyeball region cluster position vector and the preset position comparison table. The eyeball region cluster position information is compared with the facial features region feature information to determine whether the facial feature recognition model needs to be updated according to the comparison result.

[0080] The above-mentioned preset eyeball area extraction rules can be understood as the above-mentioned process of combining the position information of each part of the facial features and comprehensively matching the relative position information of the eyeball area. Specifically, the microprocessor of the present application can obtain the facial feature area feature information through the position information of the facial features contained in the facial feature information. Subsequently, multiple parts are selected for combination to obtain multiple part sets. For example, the part set corresponding to the eyeball and nose is {eyeball, nose}, the part set corresponding to the eyeball, nose, and ear is {eyeball, nose, ear}, and the part set corresponding to the eyeball, nose, mouth, ear, and eyebrows is {eyeball, nose, mouth, ear, eyebrow}. The present application can generate multiple part sets, and there is a difference in the facial features between the constituent elements of each part set.

[0081] The present application can pre-use a cluster analysis algorithm to process several preset standard facial image samples, such as a K-means clustering algorithm. Through cluster analysis processing, the association relationship between several facial parts and other irrelevant parts can be obtained, and a part association list can be generated. The association relationship can be a position association relationship between the facial parts. For example, the part set corresponding to the eyeball and the nose, by matching the part association list, obtains the part combination credibility corresponding to the eyeball at position a and the nose at position b as T1, while at this time, the part set corresponding to the eyeball, the nose, and the ear, by matching the part association list, obtains the part combination credibility corresponding to the eyeball at position a, the nose at position b, and the ear at position c as T2. The part combination credibility obtained by matching each part set is used as a vector weight, and the product and value of the vector weight and the corresponding part position vector are calculated to obtain the eyeball area cluster position vector, wherein the part position vector represents the position information of each part. The preset position comparison table contains the correspondence between different eyeball area cluster position vectors and eyeball area cluster position information. The above-mentioned eyeball area cluster position information can be obtained through the matching result of the eyeball area cluster position vector and the preset position comparison table. The position information is obtained through cluster analysis, and the feature information of the facial features is obtained through the above-mentioned CNN image processing. By comprehensively comparing the two, the accuracy of the CNN model can be judged by combining cluster analysis and CNN, thereby avoiding the problem of misidentification due to environmental factors in image acquisition, such as lighting, exposure time, etc.

[0082] Combined with the above scheme, the facial feature recognition model can be trained more flexibly to maintain the accuracy of the model. At the same time, the application can also send the comparison results to the user terminal, such as the mobile phone, computer, etc. of the medical staff, and the application does not make specific restrictions on this. The medical staff can view the comparison results and determine whether to update the facial feature recognition model, or choose manual intervention, etc.

[0083] After obtaining the position information of the eyeball area, the microprocessor will also generate a motor control instruction, which is used to call the motor to run and move the optometry lens assembly to the eyeball area.

[0084] The above technical solution is Figure 2 The eye positioning step includes extracting the eye area to obtain the coordinates of the eye feature points, the main control system obtains the coordinate information, and the motor moves to the eye.

[0085] S104, the microprocessor inputs the second image into a pre-trained eye feature extraction model to determine eye feature information of the optometry subject.

[0086] The second image is obtained by cropping the image based on the eye area position information.

[0087] The eye feature extraction model is a CNN model. The present application inputs the second image into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject, specifically including:

[0088] The image is cropped according to the eyeball area position information to obtain a second image. The second image is input into a pre-trained eye feature extraction model to determine a preset number of target feature points in the second image. The preset number of target feature points constitute the eye contour information of the optometry subject. The eye contour information at least includes multiple same-color area boundaries of the eyeball. Based on a preset plane rectangular coordinate system established with the corner of the eye as the origin, the feature point coordinates of each target feature point are determined. The coordinates of each feature point are used as eye feature information.

[0089] That is to say, the present application can crop the third image and crop the eye part from the image. The cropped second image is processed using a pre-trained eye feature extraction model to extract a preset number of target feature points in the second image. The eye feature extraction model is trained by a number of image samples with pre-labeled target feature points. Multiple target feature points are at least at the boundaries of the same color areas of the eyeball, forming the outline of the key information of the eye.

[0090] The above steps are the process of fitting eye feature points during pupil recognition.

[0091] S105, the microprocessor determines the pupil focus result corresponding to the optometry object based on the preset random forest model and eye feature information, and sends the pupil focus result to the downstream optometry task.

[0092] In an embodiment of the present application, before determining the pupil focus result corresponding to the optometry object based on the preset random forest model and the eye feature information, and sending the pupil focus result to the downstream optometry task, it also includes:

[0093] Input several eye image samples into the pre-trained eye feature extraction model to determine the feature information of each eye sample. Based on the user's annotation operation, determine the pupil center point information corresponding to each eye image. Take the feature information of each eye sample as the regression feature, take the pupil center point information corresponding to each eye sample feature information as the regression target, perform regression prediction through the random forest model to be trained, and determine the mean square error and goodness of fit corresponding to the regression prediction, so as to optimize the parameters of the random forest model to be trained based on the mean square error and goodness of fit, until the mean square error and goodness of fit meet the preset second training end condition, and obtain the preset random forest model that has completed the training.

[0094] That is to say, the present application can use the eye feature extraction model to process several eye image samples to obtain the eye sample feature information corresponding to each eye image sample, that is, the coordinates of each feature point. At the same time, the user pre-labels the pupil center coordinates for each eye image sample, that is, the pupil center information. The present application uses the eye sample feature information as a regression feature (i.e., an independent variable) and the pupil center information as a regression target (i.e., a dependent variable) to train the random forest model to be trained, and evaluates the mean square error and goodness of fit between the predicted value and the actual value. At the same time, the traversal method is used to adjust the parameters such as the number of base learners, the maximum depth, and the minimum number of samples required for splitting in the random forest. According to the situation where the mean square error and goodness of fit meet the preset second training end condition, the model with the highest prediction accuracy is selected as the prediction model in the system to obtain the preset random forest model after training. The second training end condition can be that the mean square error is less than the first preset value and the goodness of fit is greater than the second preset value. The first preset value and the second preset value are set according to the actual use situation, and the present application does not make specific restrictions on this.

[0095] The eye feature information is processed by the preset random forest model to extract the corresponding pupil center information, and the pupil center is used as the pupil focus result of the optometry object. The pupil focus result is provided to the downstream optometry task to finally generate the optometry result. The above technical solution is Figure 2 The model predicts the pupil center coordinates and accurately focuses. The downstream optometry task can be to send the pupil focus result to the user terminal, or the microprocessor can further analyze the pupil focus result to obtain the optometry result.

[0096] In addition, when the microprocessor determines that the pupil center point has changed from the previous moment, it will re-execute the above S101-S105 to update the pupil focusing result.

[0097] Through the above technical solution, the present application uses a facial feature recognition model to extract facial feature information and extract the eye area position information to achieve coarse focus of the eye area. Subsequently, the eye feature extraction model is used to extract eye feature information, and the random forest algorithm is used for regression fitting to obtain the pupil focus result, thereby achieving precise pupil focus. In this way, the eye area can be quickly located without over-reliance on manual labor, which reduces the subjective component in the optometry process, saves a lot of manpower and material resources, and can improve the user's medical experience. In addition, the present application can also handle complex image distortion and adapt to different scenarios.

[0098] Figure 3 A schematic diagram of the structure of an automatic optometry auxiliary control device provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the device includes:

[0099] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0100] After receiving the optometry preparation signal from the optometry sensing sensor, a first image is acquired from the image acquisition device in real time. The first image at least includes the face of the optometry subject. The first image is input into a pre-trained facial feature recognition model to determine the facial feature information of the optometry subject. Based on the facial feature information and the preset eyeball area extraction rules, the eyeball area position information corresponding to the optometry subject is determined, and a motor control instruction is generated according to the eyeball area position information, so that the motor drives the optometry lens assembly to move to the eyeball area matching position according to the motor control instruction. The second image is input into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject. The second image is obtained by cropping the image based on the eyeball area position information. Based on the preset random forest model and the eye feature information, the pupil focus result corresponding to the optometry subject is determined, and the pupil focus result is sent to the downstream optometry task.

[0101] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0102] After receiving the optometry preparation signal from the optometry sensing sensor, a first image is acquired from the image acquisition device in real time. The first image at least includes the face of the optometry subject. The first image is input into a pre-trained facial feature recognition model to determine the facial feature information of the optometry subject. Based on the facial feature information and the preset eyeball area extraction rules, the eyeball area position information corresponding to the optometry subject is determined, and a motor control instruction is generated according to the eyeball area position information, so that the motor drives the optometry lens assembly to move to the eyeball area matching position according to the motor control instruction. The second image is input into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject. The second image is obtained by cropping the image based on the eyeball area position information. Based on the preset random forest model and the eye feature information, the pupil focus result corresponding to the optometry subject is determined, and the pupil focus result is sent to the downstream optometry task.

[0103] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0104] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0105] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0106] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An automatic optometry auxiliary control method, characterized in that: The method comprises: After receiving the optometry preparation signal from the optometry sensing sensor, a first image is acquired from the image acquisition device in real time; wherein the first image at least includes the face of the optometry subject; Inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject; Based on the facial feature information and the preset eyeball region extraction rule, determine the eyeball region position information corresponding to the optometry object, generate a motor control instruction according to the eyeball region position information, so that the motor drives the optometry lens assembly to move to the eyeball region matching position according to the motor control instruction; Inputting the second image into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject; wherein the second image is obtained by image cropping based on the eyeball area position information; Based on the preset random forest model and the eye feature information, the pupil focus result corresponding to the optometry object is determined, so as to send the pupil focus result to the downstream optometry task.

2. The automatic optometry assisted control method according to claim 1, characterized in that: Before inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject, the method further includes: Acquire a number of image training samples and perform normalization processing; wherein the image training samples are pre-labeled with distortion labels and include corresponding distortion-free reference images; the distortion labels correspond to a set of distortion comparison parameters of the distorted images in the image training samples and the corresponding distortion-free reference images; the image training samples are acquired by a device of the same type as the image acquisition device; the image acquisition device is a wide-angle camera; Input the normalized image training samples into the first recognition sub-model to be trained until the test evaluation index corresponding to the first recognition sub-model to be trained meets the preset first training end condition, so as to obtain the trained first recognition sub-model to construct the facial feature recognition model with the second recognition sub-model; the first recognition sub-model includes an encoder and a decoder.

3. The automatic optometry assisted control method according to claim 2, characterized in that: Inputting the first image into a pre-trained facial feature recognition model to determine facial feature information of the optometry subject specifically includes: Inputting the third image after the distortion correction processing by the first recognition sub-model into the second recognition sub-model; By using the second recognition sub-model, the preset sliding window is moved in the normalized third image according to a preset order, and the pixel features in the movement coverage area are extracted by the convolution layer each time to determine whether there is facial pixel information in the movement coverage area; the facial pixel information is obtained based on the edge information and texture information corresponding to the extracted features; A facial bounding box is determined based on each facial pixel information, so as to determine the facial feature information of the optometry subject according to the position information of the facial bounding box in the third image.

4. The automatic optometry assisted control method according to claim 1, characterized in that: Based on the facial feature information and the preset eyeball area extraction rule, determining the eyeball area position information corresponding to the optometry object specifically includes: Determine feature information of facial features based on the facial feature information; wherein the feature information of facial features at least includes position information of the following parts: eyeballs, nose, mouth, ears, and eyebrows; Filtering any multiple parts in the facial feature region feature information and generating multiple part sets; each part set includes at least one part that is different from other part sets; Based on cluster analysis, a part association list is obtained by processing a number of preset standard facial feature image samples, and the relative position information of the eyeball area corresponding to each of the part sets is matched; wherein the part association list includes part position vectors under different part combinations and corresponding part combination credibility; the relative position information of the eyeball area is the part position vector of the eyeball area relative to other parts in a pre-constructed facial coordinate system; Determine the eyeball region cluster position vector according to the part combination credibility of each part set in the part association list and the weighted vector sum corresponding to the part position vector, and extract the eyeball region cluster position information according to the eyeball region cluster position vector and a preset position comparison table; The eyeball region clustering position information is compared with the facial features region feature information to determine whether the facial feature recognition model needs to be updated based on the comparison result.

5. The automatic optometry assisted control method according to claim 1, characterized in that: Inputting the second image into a pre-trained eye feature extraction model to determine the eye feature information of the optometry subject specifically includes: Performing cropping processing on the image according to the eye area position information to obtain the second image; Inputting the second image into the pre-trained eye feature extraction model to determine a preset number of target feature points in the second image; the preset number of target feature points constitute the eye contour information of the optometry subject; the eye contour information at least includes multiple same-color area boundaries of the eyeball; Determining the feature point coordinates of each of the target feature points based on a preset plane rectangular coordinate system established with the eye corner as the origin; The coordinates of each of the feature points are used as the eye feature information.

6. The automatic optometry assisted control method according to claim 1, characterized in that: Before determining the pupil focus result corresponding to the optometry object based on the preset random forest model and the eye feature information, and sending the pupil focus result to a downstream optometry task, the method further includes: Inputting a number of eye image samples into the pre-trained eye feature extraction model to determine feature information of each eye sample; Based on the user's annotation operation, determine the pupil center point information corresponding to each of the eye images; The feature information of each eye sample is used as a regression feature, and the pupil center point information corresponding to each eye sample feature information is used as a regression target. Regression prediction is performed through the random forest model to be trained, and the mean square error and goodness of fit corresponding to the regression prediction are determined, so as to perform parameter tuning on the random forest model to be trained based on the mean square error and the goodness of fit until the mean square error and the goodness of fit meet the preset second training end condition, so as to obtain the preset random forest model that has completed the training.

7. The automatic optometry assisted control method according to claim 2, characterized in that: The distortion control parameter group includes at least one or more of the following: radial distortion parameter, tangential distortion parameter, distortion center, focal length, and principal point offset.

8. The automatic optometry assisted control method according to claim 1, characterized in that: Before receiving the optometry preparation signal from the optometry sensing sensor, the method further includes: Acquire an initial optometry signal from the optometry sensing sensor; wherein the optometry sensing sensor is a thermistor; the initial optometry signal is generated when the forehead of the optometry subject contacts the thermistor; When the duration corresponding to the initial optometry signal is longer than the preset duration, the corresponding initial optometry signal is used as the optometry preparation signal; Otherwise, the initial optometry signal and the accumulated duration are discarded.

9. An automatic optometry auxiliary control device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute an automatic optometry assisted control method as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions can execute an automatic optometry assisted control method as described in any one of claims 1 to 8 above.

Images