Refractive error detection system and method

Abstract

The present application relates to a refractive error detection system and method. The refractive error detection system comprises: a plurality of lens assemblies (110), each lens assembly (110) in the plurality of lens assemblies (110) comprising a flash lamp (111) and an image capture unit arranged along an astigmatic meridian corresponding to the lens assembly (110); and a mobile terminal device (120), the plurality of lens assemblies (110) being arranged on the mobile terminal device (120), and the mobile terminal device (120) being in communication connection with the plurality of lens assemblies (110). The lens assemblies (110) are used for acquiring a target image and a target video, the target image and the target video comprising an eye region of a subject to be detected. The mobile terminal device (120) is used for inputting the target image and the target video to a refractive error detection model, and acquiring a refractive error detection result output by the model. By means of the present application, the problems of high cost and low efficiency of refractive error detection in the related art are solved.

Description

Detection system and method for refractive error

[0001] This application claims priority to U.S. Patent Application No. 18 / 751,749, filed on June 24, 2024, and Chinese Patent Application No. 202411730178.1, filed on November 28, 2024, the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] The present application belongs to the technical field of ophthalmology, and in particular relates to a detection system and method for refractive error. BACKGROUND

[0003] Globally, uncorrected refractive error is the leading cause of visual impairment and affects a large proportion of the world's population. Refractive error can first occur in infancy and typically occurs during childhood development, a critical period between 8 and 12 years of age. Despite the availability of effective treatments that can improve vision and slow the progression of myopia, a significant proportion of people do not achieve optimal visual acuity. These statistics highlight the importance of addressing the gap in eye care in terms of early diagnosis and appropriate intervention as a key issue affecting human visual health.

[0004] Therefore, there is a need for an effective method for detecting refractive error. At present, a patient can be detected by a professional such as an optometrist or an ophthalmologist using an optometry instrument or other visual detection equipment; however, such a detection method requires a professional such as an optometrist or an ophthalmologist to be professionally trained and have corresponding professional knowledge, and the optometry instrument or other visual detection equipment used is often large in size, relatively bulky and expensive, which makes the cost of detecting refractive error relatively high; at the same time, such a detection method also requires the patient to cooperate as much as possible, and for young children and infants, they often cannot relax in unfamiliar equipment and are difficult to cooperate with the optometrist or ophthalmologist, which greatly reduces the detection efficiency.

[0005] At present, there is no effective solution to the problem of high cost and low efficiency of detecting refractive error in the related art. SUMMARY

[0006] The embodiments of the present application provide a detection system and method for refractive error to at least solve the problem of high cost and low efficiency of detecting refractive error in the related art.

[0007] In a first aspect, the embodiments of the present application provide a refractive error detection system, comprising: a plurality of lens assemblies, each of the plurality of lens assemblies comprising a flash and a camera unit arranged along a meridian of astigmatism corresponding to the lens assembly; and a mobile terminal device, the mobile terminal device being provided with the plurality of lens assemblies, and the mobile terminal device being in communication connection with the plurality of lens assemblies; wherein the lens assemblies are configured to capture target images and target videos, the target images and the target videos comprising an eye region of a to-be-detected person, and the mobile terminal device is configured to input the target images and the target videos into a trained refractive error detection model, and obtain a refractive error detection result output by the refractive error detection model.

[0008] In some embodiments, the flash in at least one of the plurality of lens assemblies is an infrared light source, and the camera unit in at least one of the plurality of lens assemblies is sensitive to infrared light.

[0009] In some embodiments, the camera unit comprises a plurality of cameras, each of the cameras being configured to capture the target images and the target videos.

[0010] In some embodiments, a distance between the flash in each of the lens assemblies and each of the cameras in the lens assembly is greater than 3.5 mm.

[0011] In some embodiments, the refractive error detection model is locally stored in the refractive error detection system on a memory of the mobile terminal device.

[0012] In some embodiments, the refractive error detection model is remotely stored on a separate terminal device, and the mobile terminal device accesses the refractive error detection model through electronic connection.

[0013] In some embodiments, the mobile terminal device comprises a display and a plurality of controls, the display being configured to display the refractive error detection result, and the plurality of controls being configured to be activated by a user to activate the refractive error detection system.

[0014] In some embodiments, at least one of the plurality of controls is disposed in a preset region of the display.

[0015] In some embodiments, the display comprises a touch screen.

[0016] In some embodiments, the display is configured to display a face alignment mark, the face alignment mark being configured to instruct the user to adjust a face position of the to-be-detected person, so that the face position of the to-be-detected person is aligned with the face alignment mark.

[0017] In some embodiments, the face alignment mark comprises an eye alignment mark, the eye alignment mark being configured to instruct the user to adjust an eye position of the to-be-detected person, so that the eye position of the to-be-detected person is aligned with the eye alignment mark.

[0018] In some embodiments, the user and the to-be-detected person are the same person.

[0019] In some embodiments, the angles corresponding to at least two lens assemblies in the plurality of lens assemblies are different.

[0020] In some embodiments, the plurality of lens assemblies correspond to the same flash.

[0021] In some embodiments, the refractive error detection system further comprises a gaze indicator disposed adjacent to the at least one camera unit, such that when the lens assemblies capture the target image and the target video of the to-be-detected person, the to-be-detected person gazes at the gaze indicator.

[0022] In some embodiments, the refractive error detection result comprises at least one of one or more refractive error values along the astigmatism meridian of the left eye and the right eye, astigmatism axis, spherical power, cylindrical power, and vision screening result.

[0023] In some embodiments, the mobile terminal device is further configured to pre-process the target image and the target video, perform image quality detection on the pre-processed target image and the target video, and input the pre-processed target image and the target video to the trained refractive error detection model if the image quality of the pre-processed target image and the target video is higher than a preset index.

[0024] In a second aspect, the embodiments of the present application provide a refractive error detection method, applied to the refractive error detection system as described above, comprising: a mobile terminal device acquires a target image and a target video captured by a plurality of lens assemblies; the mobile terminal device pre-processes the target image and the target video, and performs image quality detection on the pre-processed target image and the target video; if the image quality of the pre-processed target image and the target video is higher than a preset index, the mobile terminal device inputs the pre-processed target image and the target video to a trained refractive error detection model; and the mobile terminal device acquires a refractive error detection result output by the refractive error detection model.

[0025] Compared with the related art, the embodiment of the present application provides a refractive error detection system, a plurality of lens assemblies arranged on a mobile terminal device are used to acquire a target image and a target video including an eye region of a to-be-detected person, the target image and the target video are input to a trained complete refractive error detection model by the mobile terminal device, and a refractive error detection result is acquired; an ordinary person without professional training can perform refractive error detection by himself or for others without using expensive optometry instruments or other visual detection devices, so that the detection cost is reduced; meanwhile, the refractive error detection of the to-be-detected person is relatively fast and simple by using the mobile terminal device, and the to-be-detected person does not need to specially cooperate, so that the detection efficiency can be improved. Through the present application, the problem of high detection cost and low detection efficiency of refractive error in the related art is solved, and the technical effects of reducing the detection cost of refractive error and improving the detection efficiency of refractive error are achieved.

[0026] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more clear and easy to understand. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0028] FIG. 1 is a structural schematic diagram of a refractive error detection system according to an embodiment of the present application;

[0029] FIG. 2 is a structural schematic diagram of a plurality of lens assemblies according to an embodiment of the present application;

[0030] FIG. 3 is a schematic diagram of a display according to an embodiment of the present application;

[0031] FIG. 4 is a flowchart of a refractive error detection method according to an embodiment of the present application;

[0032] FIG. 5 is a structural schematic diagram of a mobile terminal device according to an embodiment of the present application.

[0033] In the figure, the reference signs are: 100, ametropia detection system; 110, lens assembly; 111, flash; 112, first camera; 113, second camera; 114, first gaze indicator; 115, second gaze indicator; 116, infrared camera; 120, mobile terminal device; 121, display; 122, display interface; 123, processor; 124, memory; 125, computer program; 126, face alignment mark; 127, eye alignment mark; 128, distance detection control; 129, brightness detection control; 130, device angle detection control; 131, eye gaze detection control; 132, photographing / shooting option control; 133, video mode switching option control. DETAILED DESCRIPTION

[0034] In the following description, for the purposes of explanation and not limitation, specific details are set forth, such as particular system configurations, techniques, etc., in order to provide a thorough understanding of the embodiments described. However, it will be apparent to those skilled in the art that the application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the application with unnecessary detail.

[0035] It is to be understood that the terminology "includes", "has", "holds", "contains" and / or "comprising", when used in this specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0036] It is also to be understood that the terminology "and / or", when used in this specification and in the following claims, refers to one and / or all possible combinations of one or more of the associated listed items.

[0037] As used in this specification and in the claims, the term "if" can be interpreted as meaning "when", or "once", or "in response to a determination", or "in response to detecting", as appropriate, depending on the context. Similarly, the phrase "if determined", or "if detected [the described condition or event]" can be interpreted as meaning "once determined", or "in response to a determination", or "once detected [the described condition or event]", or "in response to detecting [the described condition or event]", as appropriate, depending on the context.

[0038] In addition, in the description of the application and in the following claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0039] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "including," "comprising," "having" and variations thereof are meant to encompass the items listed thereafter and equivalents thereof as well as additional items. The terms "coupled" and "connected," along with variations thereof, are used broadly and encompass both direct and indirect couplings or connections, as well as fixed and removable couplings or connections.

[0040] Globally, uncorrected refractive errors are the leading cause of visual impairment and affect a large proportion of the world's population. Refractive errors can first appear in infancy and typically develop during childhood, with the critical period being between 8 and 12 years of age. Despite the availability of effective treatments to improve vision and slow the progression of myopia, a significant proportion of people do not achieve optimal visual acuity. These statistics highlight the importance of addressing the gap in eye care in terms of early diagnosis and appropriate intervention as a key issue affecting the visual health of the human population.

[0041] Therefore, there is a need for an effective method for detecting refractive errors. The standard of care for eye and vision requires regular visits to a professional, such as an optometrist or ophthalmologist, from the age of 6 months for infants and children, and the use of a phoropter or other visual detection equipment by the professional to detect the patient; despite the existence of such community health promotion programs, due to financial constraints, low priority, and other reasons, only a small number of people take the form of regular assessment of refractive errors. In less developed or remote areas, access to eye care is more difficult and costly for most people.

[0042] In addition, professionals such as optometrists or ophthalmologists can also use a streak retinoscopy to detect refractive errors. This detection method requires a clinician with professional training and knowledge to observe and evaluate various reflection appearances to measure and identify all components of refractive errors (dioptric sphere, cylinder, and axis). Subjective refraction examination is a fine-tuning aspect of refractive error determination, which requires the patient to observe a visual target at a distance while a series of lenses are presented to the patient through a phoropter to determine whether their vision is deteriorating or improving. These detection methods also require the patient to cooperate as much as possible, and for young children and infants, they are often unable to relax in unfamiliar equipment and are difficult to cooperate with optometrists or ophthalmologists, which greatly reduces the detection efficiency.

[0043] At present, an effective solution has not been proposed for the problems of high cost and low efficiency of detection of ametropia in the related art.

[0044] Therefore, the embodiment of the present application provides an ametropia detection system. A plurality of lens assemblies arranged on a mobile terminal device are used to obtain target images and target videos including an eye region of a to-be-detected person. The mobile terminal device inputs the target images and the target videos into a trained ametropia detection model, and obtains an ametropia detection result. An ordinary person without professional training can perform ametropia detection by himself or for others without using expensive optometry instruments or other visual detection devices, thereby reducing the detection cost. Meanwhile, the use of the mobile terminal device to detect whether the eyes of the to-be-detected person are ametropic is relatively fast and simple, and the to-be-detected person does not need to specially cooperate, which can improve the detection efficiency. Through the present application, the problems of high cost and low efficiency of detection of ametropia in the related art are solved, and the technical effects of reducing the detection cost of ametropia and improving the detection efficiency of ametropia are achieved.

[0045] The ametropia detection system 100 provided by an embodiment of the present application will be described below with reference to FIG. 1. Referring to FIG. 1, FIG. 1 is a structural schematic diagram of the ametropia detection system 100 according to an embodiment of the present application. As shown in FIG. 1, the ametropia detection system 100 can include a plurality of lens assemblies 110 and a mobile terminal device 120. The plurality of lens assemblies 110 are arranged on the mobile terminal device 120 and are in communication connection with the mobile terminal device 120. Each lens assembly 110 in the plurality of lens assemblies 110 includes a flash 111 and a camera unit (including a first camera 112 and a second camera 113) arranged along a meridian line of astigmatism corresponding to the lens assembly 110. The lens assembly 110 is used to collect target images and target videos, and the target images and the target videos include an eye region of a to-be-detected person. The mobile terminal device 120 is used to input the target images and the target videos into a trained ametropia detection model, and obtain an ametropia detection result output by the ametropia detection model.

[0046] In the embodiment, the target images and the target videos can be images and videos including a face or a portrait of the to-be-detected person, images and videos including a half body of the to-be-detected person, or images and videos including a full body of the to-be-detected person.

[0047] In the embodiment, the ametropia detection model can include or partially include an image and video preprocessing algorithm, a face and eye feature detection algorithm, a face and eye region extraction algorithm, an ametropia detection algorithm, and the like.

[0048] Photorefraction is a well-recognized method for objectively screening uncorrected refractive errors, and is particularly suitable for children and developmentally disabled individuals. Among various photorefraction techniques, eccentric photorefraction is the most commonly used method because it covers a wider range of refractive errors and has a higher specificity. When the subject has a more severe refractive error, the light emitted by the flash 111 will be reflected by the retina of the subject, forming a bright red crescent-shaped light reflection (i.e., a red eye reflection).

[0049] In the embodiment, the mobile terminal device 120 can be a mobile phone, a tablet computer, a desktop computer, a laptop computer, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a cellular phone, a personal digital assistant (PDA), an augmented reality (AR) \ virtual reality (VR) device, a media player, a television, and the like. The specific form of the mobile terminal device 120 is not limited in the embodiment.

[0050] Specifically, FIG. 1(a) shows the front surface of the mobile terminal device 120, and FIG. 1(b) shows the back surface of the mobile terminal device 120. As shown in FIG. 1(a), in an embodiment, the plurality of lens assemblies 110 can be disposed on the front surface of the mobile terminal device 120. In addition, as shown in FIG. 1(b), in an embodiment, the plurality of lens assemblies 110 can also be disposed on the back surface of the mobile terminal device 120. Specifically, the plurality of lens assemblies 110 can be embedded in the front surface or the back surface of the mobile terminal device 120. When the plurality of lens assemblies 110 are disposed on the front surface of the mobile terminal device 120, the user holding the mobile terminal device 120 can perform refractive error detection in the form of self-shooting. When the plurality of lens assemblies 110 are disposed on the back surface of the mobile terminal device 120, the user holding the mobile terminal device 120 can perform refractive error detection for others or himself / herself (for example, using a selfie stick to fix the mobile terminal device 120). When the user performs refractive error detection in the form of self-shooting, the user and the subject are the same person.

[0051] It should be noted that the plurality of lens assemblies 110 can also be disposed on any surface (for example, the side surface) of the mobile terminal device 120, and the specific position of the lens assembly 110 on the mobile terminal device 120 is not limited in the embodiment.

[0052] As shown in FIG. 1, the mobile terminal device 120 comprises a display 121, which is configured to display the ametropia detection result in a preset display interface 122.

[0053] In the embodiment, the display 121 can be a display screen arranged on the front surface of the mobile terminal device 120. After the ametropia detection result is obtained by the mobile terminal device 120, the ametropia detection result can be displayed on the display interface 122 (e.g., a prompt box) of the display screen.

[0054] Specifically, the display interface 122 can comprise a text prompt area and other controls (not shown in the figure, e.g., function options such as "test again", "return to main interface", etc.). The text prompt area can display the ametropia detection result (as shown in part (a) of FIG. 1, the ametropia detection result is left eye: -0.50D, right eye: -0.50D).

[0055] In the following, the display 121 of the mobile terminal device 120 according to an embodiment of the present application will be described in conjunction with FIG. 3. As shown in FIG. 3, in one embodiment, the mobile terminal device 120 further comprises a plurality of controls configured to be activated by a user to activate the ametropia detection system 100. At least one of the plurality of controls can be arranged in a preset area of the display 121.

[0056] As an example, as shown in FIG. 3, the preset area of the display 121 comprises two types of controls. The first type of control is a detection indication control, which comprises a distance detection control 128, a brightness detection control 129, a device angle detection control 130, and an eye gaze detection control 131. The second type of control is a function option control, which comprises a photographing / shooting option control 132 and a video mode switching option control 133.

[0057] Specifically, the plurality of detection indication controls can be used to ensure that a plurality of requirements for performing the ametropia detection are met, thereby ensuring that the quality of the obtained target images and target videos is high. As shown in FIG. 3, in the case where the related requirements are met, the detection indication control corresponding to the met related requirement can turn green, while the other detection indication controls turn red (for example, the distance detection control 128 and the brightness detection control 129 in FIG. 3 turn green because the preset distance requirement and the brightness requirement are met, while the device angle detection control 130 and the eye gaze detection control 131 turn red (represented in the form of shading) because the preset device angle requirement and the eye gaze requirement are not met).

[0058] The function option control is configured to satisfy multiple functions, for example, the user can operate the video mode switching option control 133 to switch to a photographing mode (in which a target image of the to-be-detected person is acquired) or a video shooting mode (in which a target video of the to-be-detected person is acquired); the user can operate the photographing / video shooting option control 132 to determine when to acquire the target image and the target video of the to-be-detected person.

[0059] As an example, before entering the display interface 122 displayed by the display 121 shown in FIG. 3, other function option controls can also be included, for example, a language selection control, a return control, a main menu control, and the like.

[0060] As shown in FIG. 3, in one embodiment, the display 121 is configured to display a face alignment mark 126, which is used to instruct the user to adjust the face position of the to-be-detected person so that the face position of the to-be-detected person is aligned with the face alignment mark 126. The face alignment mark 126 can include an eye alignment mark 127, which is used to instruct the user to adjust the eye position of the to-be-detected person so that the eye position of the to-be-detected person is aligned with the eye alignment mark 127.

[0061] In the present embodiment, the user can adjust the face position and the eye position of the to-be-detected person based on the face alignment mark 126 and the eye alignment mark 127, so that the face of the to-be-detected person is in the correct direction and the distance between the eyes of the to-be-detected person and the mobile terminal device 120 is appropriate, thereby ensuring that the acquired target image and target video have high quality and further improving the accuracy of subsequent ametropia detection.

[0062] In one embodiment, the display 121 can include a touch screen, and the user can perform a touch or press operation on a corresponding region on the touch screen to select a corresponding control to activate the ametropia detection system 100.

[0063] In one embodiment, the ametropia detection result displayed by the display interface 122 can include at least one of one or more ametropia values of the left eye and the right eye of the to-be-detected person along the astigmatism meridian, an astigmatism axis position, a spherical power, a cylindrical power, and a vision screening result.

[0064] When the ametropia detection result includes the above data, the text prompt area of the display interface 122 also displays the above data. The vision screening result can include a “qualified / failed” detection result, a “positive / negative” detection result, and the like, which are used to represent whether the to-be-detected person has ametropia. For example, if the ametropia detection result displayed by the display interface 122 is the vision screening result-qualified or the vision screening result-negative, it indicates that the to-be-detected person does not have ametropia.

[0065] In addition, the refractive error detection result can also include various confidence scores and / or an explanation corresponding to the detection result and / or a visual model corresponding to the detection result, etc.

[0066] In one embodiment, each lens assembly 110 can include a flash 111 and a camera unit arranged along the astigmatism meridian corresponding to the lens assembly 110. The camera unit can include a plurality of cameras, each of which is used to capture target images or target videos. As an example, the camera unit shown in FIG. 1 includes a first camera 112 and a second camera 113. It should be noted that the number of cameras included in the camera unit can not be limited to two, but can also be three, four or even more, and the present application does not limit the number of cameras included in the camera unit.

[0067] In the present embodiment, the angles of the astigmatism meridians corresponding to at least two lens assemblies 110 in the plurality of lens assemblies are different. Specifically, the astigmatism meridian corresponding to each lens assembly 110 can include one of the following: a 180° astigmatism meridian, a 135° astigmatism meridian, and a 90° astigmatism meridian.

[0068] As an example, as shown in FIG. 1, in order to obtain more accurate refractive error detection results, at least three target images or target videos can be obtained from different directions, so the front surface (see FIG. 1(a)) and the back surface (see FIG. 1(b)) of the mobile terminal device 120 can be respectively provided with three lens assemblies 110, and the astigmatism meridians corresponding to the three lens assemblies 110 are respectively a 180° astigmatism meridian, a 135° astigmatism meridian, and a 90° astigmatism meridian.

[0069] In the present embodiment, by providing three lens assemblies 110 on the front surface and the back surface of the mobile terminal device 120, and the astigmatism meridians corresponding to the three lens assemblies 110 are respectively a 180° astigmatism meridian, a 135° astigmatism meridian, and a 90° astigmatism meridian, the refractive error of the subject to be detected can be detected from multiple astigmatism meridians. Among them, the refractive error along at least three astigmatism meridians (180° astigmatism meridian, 135° astigmatism meridian, and 90° astigmatism meridian) can be determined using the following calculation formula: Fθ=C×sin2|θ–A|+S;

[0070] Wherein, Fθ is the degree of refractive error of the subject to be detected measured along the astigmatism meridian θ (which can be in radians), S is the spherical refractive power, C is the cylindrical component, and A is the astigmatism axis.

[0071] In this way, the flash 111, the first camera 112, and the second camera 113 in the three groups of lens assemblies 110 are arranged along the 180° astigmatism meridian (horizontal meridian), the 135° astigmatism meridian (oblique meridian), and the 90° astigmatism meridian (vertical meridian), respectively, and the range of refractive errors detected by the refractive error detection system 100 should be within the range of +8.00D to -8.00D.

[0072] It should be noted that although the front surface and the rear surface of the mobile terminal device 120 are each provided with three groups of lens assemblies 110 in FIG. 1, more or fewer lens assemblies 110 can be provided on the mobile terminal device 120, for example, two groups of lens assemblies 110 can be provided on the front surface of the mobile terminal device 120, and four, five, or the like groups of lens assemblies 110 can be provided on the rear surface of the mobile terminal device 120, and the present application does not limit the specific positions and the number of lens assemblies 110 provided on the mobile terminal device 120.

[0073] It should be noted that the flash 111, the first camera 112, and the second camera 113 (collectively referred to as the camera unit) in the lens assembly 110 can not only be arranged along the 180° astigmatism meridian, the 135° astigmatism meridian, or the 90° astigmatism meridian, but also can be arranged along other astigmatism meridians, for example, along the 30° astigmatism meridian, the 45° astigmatism meridian, the 60° astigmatism meridian, the 90° astigmatism meridian, the 91° astigmatism meridian, the 120° astigmatism meridian, the 140° astigmatism meridian, the 270° astigmatism meridian, and the like, so that the refractive error detection system 100 can detect refractive errors along more astigmatism meridians, thereby expanding the detection range and the application range of the refractive error detection system 100.

[0074] In one embodiment, the flash 111 in at least one of the groups of lens assemblies 110 is an infrared light source, and the camera unit in at least one of the groups of lens assemblies 110 is sensitive to infrared light.

[0075] As an example, the flash 111 shown in FIG. 1 can be a flash 111 emitting visible light or an infrared flash 111 emitting infrared light. When the flash 111 is a flash 111 emitting visible light, the refractive error detection system 100 can collect target images and target videos with higher image quality; when the flash 111 is an infrared flash 111, the lens unit corresponding to the infrared flash 111 can be sensitive to infrared light, for example, the lens unit can include an infrared camera, which can minimize the pupil constriction effect affecting the size of the retinal reflection, so that the crescent-shaped image formed by the retinal reflection of the to-be-detected person can more accurately reflect the refractive error level of the to-be-detected person, thereby improving the detection accuracy of the refractive error.

[0076] In one embodiment, the flash 111 corresponding to the plurality of lens assemblies 110 can be the same flash 111.

[0077] As an example, as shown in part (b) of FIG. 1, the flash 111 corresponding to the three lens assemblies 110 can be the same flash 111.

[0078] In one embodiment, the distance between the flash 111 and each camera in the lens assembly 110 in each group of lens assemblies 110 is greater than 3.5 mm.

[0079] In the present embodiment, the distance between the flash 111 and each camera in the lens assembly 110 in each group of lens assemblies 110 is designed according to a data simulation model, which covers the image acquisition working distance for low, medium, and high ranges of refractive errors and natural un-dilated pupils. It should be noted that although the data simulation model is based on natural, un-dilated pupils, the refractive error detection system 100 provided in the present embodiment can also be applied to dilated pupils, i.e., after the pupil of the subject to be detected is dilated by using cycloplegic diagnostic eye drops to limit the contraction of the pupil of the subject to be detected to the light source, the refractive error detection system 100 provided in the present embodiment can be used to detect the refractive error of the subject to be detected.

[0080] Specifically, the size of the refractive error can be obtained by the following calculation formula:

[0081] For myopia: Rx = e / [d(2r-s)] + D;

[0082] For hyperopia: Rx = e / [d(2r-s)] - D;

[0083] wherein Rx is the refractive power, s is the size of the crescent, 2r is the pupil diameter, D is the working distance of the refractive error monitoring system, and e is the eccentricity of the flash 111 (i.e., the distance between the flash 111 and the camera).

[0084] From the above calculation formula, the optimal eccentricity of the flash 111 can be obtained under the condition that the average non-dilated pupil diameter is at least 2 millimeters (mm) and the refractive error measurement range is for myopia and hyperopia. For example, when the working distance D is 3 meters (m), the collected image should be able to clearly record one or both eyes, and the iris, pupil, and crescent profile can be clearly observed therefrom.

[0085] However, in the current mobile terminal device 120 with built-in flash 111 and camera, the camera is always placed very close to the flash 111, which results in a low e value. If the eye of the subject to be detected is in a low range of refractive error (+3.00D to -3.00D), the crescent-shaped image formed by the reflection of the retina of the subject to be detected is often invisible, which results in s being null. According to the above calculation formula, when the position of the camera is further moved (as a function of the variable "e") and the refractive power is maximized and minimized (myopia and hypermetropia Rx), it can be obtained that "e" is greater than 3.5 mm. Therefore, by setting the distance between the flash 111 in each lens assembly 110 and each camera in the lens assembly 110 to be greater than 3.5 mm, the best eccentricity for lens-to-light positioning can be provided.

[0086] It should be noted that the distance between the flash 111 in each lens assembly 110 and each camera in the lens assembly 110 is designed to be greater than 3.5 mm based on the current application scenario and user demand. In different application scenarios (e.g., requiring more accurate refractive error detection results) and user demands, the distance between the flash 111 in each lens assembly 110 and each camera in the lens assembly 110 can be designed to be much higher than 3.5 mm or less than 3.5 mm, or the distance between the flash 111 in each lens assembly 110 and each camera in the lens assembly 110 can be designed to be greater than their minimum physical size.

[0087] In one embodiment, the refractive error detection system 100 further comprises a gaze indicator, which is arranged adjacent to at least one camera unit, so that when the lens assembly captures the target image and target video of the subject to be detected, the subject to be detected gazes at the gaze indicator.

[0088] As an example, please refer to FIG. 2, which is a structural schematic diagram of a plurality of lens assemblies 110 according to an embodiment of the present application. A color indicator 114 and / or a frame indicator 115 can be arranged around at least one of the first camera 112 and the second camera 113 in each lens assembly 110.

[0089] In the present embodiment, by arranging the color indicator 114 and / or the frame indicator 115 around the camera, a color recognition mark can be formed when capturing the target image and target video of the subject to be detected. The color recognition mark can serve as a distinguishable gaze mark of the subject to be detected. In the subsequent target image and target video processing process, it can be determined whether the subject to be detected is gazing at the camera according to the distinguishable gaze mark.

[0090] As shown in FIG. 2, in one embodiment, the lens assembly 110 can further include an infrared camera 116 or other detachable accessories. Among others, the infrared camera 116 can further assess the refractive errors along various astigmatic meridians and control the pupillary reflex.

[0091] In the present embodiment, the first camera 112, the second camera 113 and the infrared camera 116 in the lens assembly 110 can be used for vision and eye care related applications, such as detection, identification and monitoring of refractive errors, and eye biometry of one or both eyes.

[0092] In the related art, there are some methods that can use mobile terminal devices with built-in flashlights and cameras as portable eccentric photography refraction tools (e.g., GoCheck Kids, an application that allows pediatricians to perform photo screening tests without the need to purchase a dedicated pediatric vision screening device), however, the detection range of these methods.

[0093] And, when using these methods, the subject to be detected needs to use cycloplegic diagnostic eye drops to limit the contraction of the subject's pupil to the light source, so that the subject's pupil is dilated to collect the crescent-shaped image on the subject's eye, therefore, ordinary people without professional training still cannot accurately use this detection method.

[0094] In addition, using such methods to obtain visible crescent-shaped images formed by the retina reflection of the subject to be detected will limit the range of refractive errors that can be detected; if the eye of the subject to be detected is in a lower refractive error range (+3.00D to -3.00D), the crescent-shaped image formed by the retina reflection of the subject to be detected is often invisible, and the image processor and filter in such mobile terminal devices will also over-process many areas of the image, making the visible crescent-shaped image invisible, therefore, the application value of such detection methods in accurate reading is limited, especially in detecting hyperopia, astigmatism and low refractive error range.

[0095] The embodiment of the present application provides a refractive error detection system, a plurality of lens assemblies arranged on a mobile terminal device are used to acquire target images and target videos including eye regions of a to-be-detected person, the mobile terminal device inputs the target images and the target videos into a trained complete refractive error detection model, and a refractive error detection result is acquired; an ordinary person without professional training can perform refractive error detection by himself or for others without using expensive optometry instruments or other visual detection devices, so that the detection cost is reduced; meanwhile, the mobile terminal device is used to detect whether the eyes of the to-be-detected person are refractive, which is relatively fast and simple, and the to-be-detected person does not need to specially cooperate, so that the detection efficiency can be improved. Through the present application, the problem of high detection cost and low detection efficiency of refractive error in the related art is solved, and the technical effects of reducing the detection cost of refractive error and improving the detection efficiency of refractive error are achieved.

[0096] In one embodiment, after the lens assembly 110 acquires the target images and the target videos, the mobile terminal device 120 can further pre-process the target images and the target videos, perform image quality detection on the pre-processed target images and target videos, and input the pre-processed target images and target videos into the trained complete refractive error detection model when the image quality of the pre-processed target images and target videos is higher than a preset index.

[0097] In the embodiment, the mobile terminal device 120 can further pre-process the target images and the target videos, and extract images and videos (also referred to as eye images and eye videos) corresponding to the eye regions of the to-be-detected person in the target images and the target videos.

[0098] In this embodiment, the lens assembly 110 can capture still images, videos or 3D images of the subject to be detected at different wavelengths (e.g. visible light or near / central infrared light). These still images, videos or 3D images can include images of the subject to be detected’s eyes, head, half body, full body or other images, and the mobile terminal device 120 can extract the subject to be detected’s eye images and eye videos from them using image processors and in combination with artificial intelligence recognition, and these operations can be triggered automatically, semi-automatically and manually at any frequency, any number of times and any time conditions, whether or not any prompt / reminders are provided to the user. In addition, in addition to the lens assembly 110, image capture devices (all types of smartphones (including all brands) (with or without flash 111) (one or more built-in or external flash 111), all types of cameras (mirror reflex cameras, digital cameras, point-and-shoot cameras, action cameras and all other types of cameras) (with or without flash 111) (one or more built-in or external flash 111), all types of tablets (including all brands) (with or without flash 111) (one or more built-in or external flash 111) and all other image capture devices (with or without flash 111) (one or more built-in or external flash 111)) in communication with the mobile terminal device 120 can be used to obtain target images and target videos.

[0099] In addition, target images and target videos of the subject to be detected can be captured at any distance (distance between the image capture device and the subject to be detected) using any triggering method (e.g. Wi-Fi, Bluetooth and short signals), and any AI adjustment for improving the accuracy of the capture can be used, such as distance adjustment and feedback, stability adjustment and feedback, and user lighting adjustment and feedback. The image capture device can also be fixed to a strap or other mechanism on the user’s and / or robot system’s hand or other component and / or adapted to unique environmental changes, such as but not limited to temperature control of the device and / or gravity tilt during operation, which can be adjusted and designed to facilitate easy and accurate positioning and stabilization of the human hand and / or robot hand.

[0100] After obtaining the target images and target videos, the target images and target videos can be converted into digital format, modulated into any form of transmittable wave and sent to the trained complete ametropia detection model (machine / deep learning and / or other artificial intelligence algorithms / models). This includes all types of transmission using any type of device and / or imaging instrument, such as phone to cloud, phone to laptop, phone to local / external server, etc. This also includes transmission and / or file loading occurring within a single device, such as the mobile terminal device 120 can transmit / load the target images and target videos to the ametropia detection model built-in and running in the mobile terminal device 120.

[0101] In the above embodiments, the ametropia detection model is described as being stored locally on the storage of the mobile terminal device 120 in the ametropia detection system 100. In this case, the ametropia detection system 100 on the storage of the mobile terminal device 120 can still function normally (i.e., interact normally with the ametropia detection model) without relying on an Internet connection, even if the mobile terminal device 120 does not have access to the Internet.

[0102] In other embodiments, the ametropia detection model can also be stored remotely on a separate terminal device, and the mobile terminal device 120 accesses the ametropia detection model through an electronic connection.

[0103] The mobile terminal device 120 can perform image preprocessing on the target images and target videos, which includes all types of data preprocessing and / or image preprocessing, such as based on signal processing (i.e., Gaussian transform, Laplace transform, etc.) and / or based on machine learning and / or based on deep learning (e.g., super resolution based on deep learning models) and other methods during model running (e.g., adjusting image size).

[0104] After preprocessing the target images and target videos, the mobile terminal device 120 can perform artificial intelligence-based image quality detection on the preprocessed target images and target videos, and input the preprocessed target images and target videos to the trained ametropia detection model if the image quality of the preprocessed target images and target videos is higher than the preset index; otherwise, the mobile terminal device 120 can pop up a prompt box (e.g., including “picture not clear enough, please retake”) on the display interface 122 of the display 121 to remind the user to reacquire the target images and target videos. In this way, the target images and target videos with poor image quality can be effectively prevented from being input to the ametropia detection model, thereby improving the detection accuracy of the ametropia detection model.

[0105] Specifically, the image quality detection can include detection methods for the human body, face, and eyes, as well as detection of any elements of the eyes, such as detection of the iris, pupil, and crescent, and also include image cropping methods with or without the use of artificial intelligence technology.

[0106] In one embodiment, the AI-based image quality detection can include at least one of the following: A. Eye area coverage (EAC): area of eye region / total image; B. Valid iris coverage (VIC): unobstructed iris region / total iris region; C. Relative pupil radius (RPR): pupil radius / iris radius; D. Brightness of eye area (BEA): average light intensity within eye region; E. Deviation of iris area (DIA): horizontal distance between iris center and eye center; F. Score of sharpness (SOS): measure whether the image is blurry (variance based on gradient).

[0107] According to the above multiple indicators, the image quality of the preprocessed target image and target video is detected, which can evaluate the quality of the preprocessed target image and target video in multiple aspects whether to reach the input standard of the ametropia detection model. In the case that the image quality of the preprocessed target image and target video is higher than the preset indicators, it is input to the ametropia detection model, which can improve the data quality input to the ametropia detection model, thereby improving the detection accuracy of the ametropia detection model. It should be noted that the preset indicators can be determined according to the specific type of image quality detection, user expectation and actual application scenario, which is not limited in the present application.

[0108] In one embodiment, the ametropia detection model can be trained based on machine / deep learning and other artificial intelligence algorithms / models. The training method can include all types of machine learning methods (supervised, unsupervised, etc.) and / or deep learning methods for predicting ametropia. The feature extraction method can use any type of feature engineering and selection method, including any engineering and extraction of eye element (i.e. iris, pupil, crescent, etc.) related features, such as appearance-based features (e.g. size of iris, pupil, crescent, etc.). In addition, any programming language (including but not limited to python, objective-C, C / C++, Java, etc.) and / or any machine / deep learning platform (e.g. Pytorch, TensorFlow, Caffe, etc.) can be used to implement the construction of the ametropia detection model.

[0109] The construction of the ametropia detection model can also include any method of modeling one image using only one model, and / or any method of jointly and / or separately modeling any number of images using any number of models. The ametropia detection model can include a framework with one or more model stems / branches for processing any number of input images and input videos (e.g., target images and target videos taken at one or more astigmatism meridians (e.g., target images and target videos taken at 180° astigmatism meridian, 135° astigmatism meridian, and 90° astigmatism meridian)).

[0110] The output of the ametropia detection model can include at least one of one or more ametropia values of the left eye and the right eye of the to-be-detected person along the astigmatism meridian, astigmatism axis, spherical power, cylindrical power, and vision screening results. In addition, the output of the ametropia detection model can also include various confidence scores and / or explanations corresponding to the detection results and / or visualization models corresponding to the detection results, etc.

[0111] During the training process of the ametropia detection model, a data classification method can be used so that each ametropia range in the training data set includes at least 20 training images, thereby ensuring that each ametropia range has a corresponding number of approximately equal training data sets, thereby strengthening the balanced training between each ametropia range; synthetic training images can also be created based on existing few data pools, thereby perfecting the data pool with the least number of training images. Compared with the "random" data set training method, this training optimization method can improve the output accuracy of the ametropia detection model.

[0112] In addition, since there are overlapping features between most emmetropia ranges and high ametropia ranges, specific model data can be identified according to the unique features of each ametropia range (e.g., lower light intensity in images with high ametropia). Compared with the use of a "random" data set training method, these new training images are included in the training data set, and the ametropia detection model is trained with this training data set, which can enhance the ability of the ametropia detection model to identify these features while not affecting the original overlapping features between emmetropia ranges and high ametropia ranges.

[0113] The ametropia detection method provided by an embodiment of the present application will be described below in combination with the drawings. The ametropia detection method is applied to the ametropia detection system 100 provided by the above-mentioned embodiments of the present application. Please refer to FIG. 4, which is a flowchart of the ametropia detection method according to an embodiment of the present application. As shown in FIG. 4, the method includes steps 401 to 404:

[0114] At step 401, the mobile terminal device 120 acquires the target image and the target video collected by the multiple groups of lens assemblies 110.

[0115] At step 402, the mobile terminal device 120 pre-processes the target image and the target video, and performs image quality detection on the pre-processed target image and the target video.

[0116] At step 403, in the case that the image quality of the pre-processed target image and the target video is higher than a preset index, the mobile terminal device 120 inputs the pre-processed target image and the target video into the trained complete ametropia detection model.

[0117] At step 404, the mobile terminal device 120 acquires the ametropia detection result output by the ametropia detection model.

[0118] In this embodiment, the ametropia detection method can be implemented by the ametropia detection system 100 provided in the above-mentioned embodiments of the present application, or can be implemented by an external terminal device coupled with the ametropia detection system 100 provided in the above-mentioned embodiments of the present application, which is not limited in the present application.

[0119] It should be noted that the detailed content of each step above is based on the same concept as the system embodiments of the present application, and the specific functions and technical effects brought about can be referred to the system embodiments part, which will not be repeated here.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is taken as an example for illustration, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or software. In addition, the specific names of each functional unit and module are only for easy distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the above system can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0121] FIG. 5 is a structural schematic diagram of a mobile terminal device according to an embodiment of the present application. As shown in FIG. 5, the mobile terminal device 120 includes, in addition to the display 121, at least one processor 123 (only one is shown in FIG. 5), a memory 124, and a computer program 125 stored in the memory 124 and executable on the at least one processor 123, and the processor 123 implements the steps in any of the above refractive error detection method embodiments when executing the computer program 125.

[0122] The mobile terminal device 120 can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The mobile terminal device 120 can include, but is not limited to, the processor 123 and the memory 124. Those skilled in the art can understand that FIG. 5 is only an example of the mobile terminal device 120, and does not constitute a limitation on the mobile terminal device 120, and the mobile terminal device 120 can include more or fewer components than those shown, or combine certain components, or different components, for example, can also include an input / output device, a network access device, and the like.

[0123] The processor 123 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0124] The memory 124 can be an internal storage unit of the mobile terminal device 120, such as a hard disk or a memory of the mobile terminal device 120 in some embodiments. The memory 124 can also be an external storage device of the mobile terminal device 120, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the mobile terminal device 120 in other embodiments. In other embodiments, the memory 124 can include both an internal storage unit and an external storage device of the mobile terminal device 120. The memory 124 is used to store an operating system, an application program, a BootLoader, data, and other programs, such as program codes of the computer program 125, etc. The memory 124 can also be used to temporarily store data that has been output or is to be output.

[0125] The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in the ametropia detection method embodiments.

[0126] The computer program product is run on the mobile terminal, and the computer program product enables the mobile terminal to implement the steps in the ametropia detection method embodiments.

[0127] The computer program includes computer program codes, and the computer program codes can be in a source code form, an object code form, an executable file, or some intermediate form, etc. The computer readable medium at least includes any entity or device capable of carrying the computer program codes to the mobile terminal device 120, a recording medium, a computer memory, a Read-Only Memory (ROM), a Random Access Memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc.

[0128] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0129] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0130] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely schematic, for example, the division of the modules or units is merely a logical function division, and there can be another division in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0131] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A refractive error detection system, comprising: Multiple lens assemblies, each of the multiple lens assemblies including a flash and a camera unit arranged along the astigmatic meridian corresponding to the lens assembly; as well as A mobile terminal device, wherein the mobile terminal device is provided with the plurality of lens assemblies, and the mobile terminal device is communicatively connected to the plurality of lens assemblies; The lens assembly is used to acquire target images and target videos, including the eye area of ​​the subject to be detected. The mobile terminal device is used to input the target images and target videos into a fully trained refractive error detection model and obtain the refractive error detection results output by the refractive error detection model.

2. The refractive error detection system according to claim 1, wherein, The flash in at least one of the multiple lens assemblies is an infrared light source, and the camera unit in at least one of the multiple lens assemblies is sensitive to infrared light.

3. The refractive error detection system according to claim 1, wherein, The camera unit includes multiple cameras, each of which is used to capture target images and target videos.

4. The refractive error detection system according to claim 3, wherein, The distance between the flash in each lens assembly and each camera in the lens assembly is greater than 3.5 mm.

5. The refractive error detection system according to claim 1, wherein, The refractive error detection model is locally stored in the refractive error detection system on the memory of the mobile terminal device.

6. The refractive error detection system according to claim 1, wherein, The refractive error detection model is remotely stored on a separate terminal device, and the mobile terminal device accesses the refractive error detection model via an electronic connection.

7. The refractive error detection system according to claim 1, wherein, The mobile terminal device includes a display and multiple controls. The display is used to show the refractive error detection results; The multiple controls are used by the user to activate the refractive error detection system.

8. The refractive error detection system according to claim 7, wherein, At least one of the plurality of controls is set in a preset area of ​​the display.

9. The refractive error detection system according to claim 7, wherein, The display includes a touchscreen.

10. The refractive error detection system according to claim 7, wherein, The display is used to show a face alignment indicator, which instructs the user to adjust the face position of the subject to be tested so that the face position of the subject to be tested is aligned with the face alignment indicator.

11. The refractive error detection system according to claim 10, wherein, The facial alignment marker includes an eye alignment marker, which instructs the user to adjust the position of the subject's eyes so that the subject's eyes are aligned with the eye alignment marker.

12. The refractive error detection system according to claim 8 or 9, wherein, The user and the person to be tested are the same person.

13. The refractive error detection system according to claim 1, wherein, At least two of the multiple lens assemblies correspond to different angles.

14. The refractive error detection system according to claim 1, wherein, The flash units corresponding to the multiple lens assemblies are the same flash units.

15. The refractive error detection system of claim 1 further includes a gaze indicator disposed adjacent to at least one of the camera units such that the subject gazes at the gaze indicator when the lens assembly acquires the target image and target video of the subject.

16. The refractive error detection system according to claim 1, wherein, The refractive error detection results include at least one of the following: one or more refractive error values, astigmatic axis, spherical power, cylindrical power, and vision screening / observation results for the left and right eyes along the astigmatic meridian.

17. The refractive error detection system according to claim 1, wherein, The mobile terminal device is also used to preprocess the target image and target video, perform image quality detection on the preprocessed target image and target video, and input the preprocessed target image and target video into a fully trained refractive error detection model if the image quality of the preprocessed target image and target video is higher than a preset index.

18. A method for detecting refractive errors, applied to the refractive error detection system as described in any one of claims 1 to 17, comprising: The mobile terminal device acquires target images and target videos captured by the multiple sets of lens components; The mobile terminal device preprocesses the target image and target video, and performs image quality detection on the preprocessed target image and target video; If the image quality of the preprocessed target image and target video is higher than the preset index, the mobile terminal device inputs the preprocessed target image and target video into the fully trained refractive error detection model. The mobile terminal device acquires the refractive error detection results output by the refractive error detection model.

Images