Eye image recognition method based on multi-task learning and related device

Abstract

This application proposes an eye image recognition method and related equipment based on multi-task learning. In order to achieve cataract screening while performing refractive error screening, without increasing additional investment or affecting the refractive error screening process, a multi-task image recognition model is proposed based on datasets from different fields for multi-task learning. This model is applicable to refractive error recognition, pupil segmentation, and cataract recognition tasks. Compared with single-task image recognition models, it improves image recognition performance. Thus, after obtaining the eye image to be identified, this application can directly process the eye image based on this multi-task image recognition model and output the task reference data of the eye to be identified in each of the three tasks. This effectively assists doctors in quickly and more accurately determining the task recognition results of the eye to be identified in different fields.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to an eye image recognition method and related equipment based on multi-task learning. Background Technology

[0002] When the eye views an external object, light emitted or reflected from the object enters the eye's refractive system (such as the cornea and lens), is refracted, and then converges to form an image on the retina. Therefore, the degree of refractive error in the eye determines the clarity of the image focused on the retina. If the refractive media undergoes deformation and / or clouding, the observed object will be distorted, blurred, or even invisible, severely impacting the patient's daily life.

[0003] Therefore, refractive testing has become an important part of various ophthalmic diseases such as cataracts, glaucoma, diabetes, and refractive errors. Doctors usually determine whether a patient has refractive errors, cataracts, or other ophthalmic diseases by observing the images of the patient's eyes collected by an eccentric radiography device and combining them with clinical experience. This requires a high level of professional expertise from the doctor and is prone to misdiagnosis due to human error. Summary of the Invention

[0004] To address the above problems, this application proposes the following technical solution:

[0005] On the one hand, this application proposes an eye image recognition method based on multi-task learning, the method comprising:

[0006] Obtain an image of the eye to be identified;

[0007] Based on the trained multi-task image recognition model, the eye image to be identified is processed to obtain task reference data for multiple tasks in different domains of the eye to be identified; wherein, the multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different domains, the datasets in different domains include different eye images with corresponding task labels, and the multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task.

[0008] Output task reference data for each of the multiple tasks to assist in determining the task recognition results of the eye to be identified in different domains.

[0009] Optionally, obtaining the datasets from the different domains includes:

[0010] Original eye images of multiple sample eyes were acquired; the original eye images were obtained by an eccentric photorefractive device.

[0011] The original eye image is analyzed to obtain the refractive data corresponding to the sample eye, and the refractive data is determined as the refractive power task label of the original eye image; the refractive data includes spherical power and cylindrical power;

[0012] The original eye image is segmented to obtain the pupil mask image and pupil positioning information corresponding to the sample eye. The pupil positioning information is then used as the segmentation task label for the original eye image.

[0013] Acquire lens opacity data and / or cataract category information for each of the multiple sample eyes, and determine the lens opacity data and / or the cataract category information as the cataract task label for the corresponding original eye image; the lens opacity data and the cataract category information are determined by observation results of the corresponding sample eyes using a slit-lamp microscope;

[0014] Data sets from different domains are obtained using the original eye image labeled with the refractive power task label, the pupil mask image labeled with the segmentation task label and the refractive power task label, and the original eye image labeled with the cataract task label.

[0015] Optionally, the multi-task image recognition model includes a shared encoder and decoders corresponding to multiple tasks belonging to different domains;

[0016] The process involves processing the eye image to be identified based on a trained multi-task image recognition model to obtain task reference data for multiple tasks in different domains, including:

[0017] The image to be identified is input into the shared encoder for encoding processing to obtain shared feature data for the multiple tasks; the shared feature data can characterize the hidden relationships between the multiple tasks;

[0018] The shared feature data is input into the decoders corresponding to the multiple tasks for processing to obtain the task reference data of the eye to be identified for the multiple tasks in different fields.

[0019] The network structures of the decoders corresponding to the multiple tasks are different, and the network structure of the decoders is determined according to the corresponding task type.

[0020] Optionally, the step of inputting the image to be recognized into the shared encoder for encoding processing to obtain shared feature data for the multiple tasks includes:

[0021] The image to be identified is input into the shared encoder, and features are extracted through a convolutional layer of the first scale. After processing by a max pooling layer, a first feature map is obtained.

[0022] The first feature map is sequentially input into the second-scale convolutional layer and the first-scale convolutional layer for feature extraction. After processing by the max pooling layer, the second feature map is obtained.

[0023] The second feature map is sequentially input into two convolutional layers of the first scale for feature extraction. The third feature map is then processed by a max pooling layer to obtain shared feature data for the multiple tasks.

[0024] In this configuration, the convolution kernels in the first-scale convolutional layer perform convolution operations according to a first step length, and the convolution kernels in the second-scale convolutional layer perform convolution operations according to a second step length. The first step length is smaller than the second step length, and the first scale of the convolution kernel is smaller than the second scale.

[0025] Optionally, the method further includes:

[0026] Determine whether the image size of the eye image to be identified meets the input image size requirements of the shared encoder;

[0027] If it does not meet the requirements, the eye image to be identified shall be processed according to the input image size requirements;

[0028] The step of inputting the image to be recognized into the shared encoder includes:

[0029] The processed eye image that meets the input image size requirements is input into the shared encoder.

[0030] Optionally, the step of inputting the shared feature data into the decoders corresponding to the multiple tasks for processing to obtain task reference data for the eye to be identified in the multiple tasks in different domains includes:

[0031] The shared feature data is input into a regression decoder for the refractive error recognition task. Through regression processing with two fully connected layers, refractive reference data of the eye to be identified is obtained. The refractive reference data includes spherical power and cylindrical power, which characterize the refractive error of the eye to be identified.

[0032] The shared feature data is input into a classification decoder for the cataract identification task. After classification processing through two fully connected layers, the classification results are normalized to obtain the predicted probability that the eye to be identified belongs to the cataract category, and / or the lens opacity data of the eye to be identified; the lens opacity data can characterize the cataract level of the eye to be identified.

[0033] The shared feature data is input into the segmentation decoder for the pupil segmentation task. After feature extraction through convolutional layers of different scales, the feature maps output by the connected convolutional layers are processed through multiple upsampling layers or feature joint layers. The processed feature maps are then input into a unit convolutional layer for feature data processing to obtain the pupil region mask of the eye to be identified.

[0034] The convolutional layers of different scales include multiple convolutional layers of the first scale and one convolutional layer of the second scale; the feature fusion layer is connected to the corresponding convolutional layer of the shared encoder to realize the feature fusion operation between the feature map output by the convolutional layer and the feature map output by the convolutional layer connected in the segmentation decoder, so as to increase the low-order feature data lost by the feature extraction process of the convolutional layer connected in the segmentation decoder.

[0035] Optionally, the step of performing multi-task learning based on datasets from different domains to obtain a multi-task image recognition model includes:

[0036] Images from datasets in different domains are input into an initial shared encoder for feature extraction, resulting in first-domain feature data and second-domain feature data.

[0037] Minimize the distribution difference processing of the first domain feature data and the second domain feature data to obtain sample feature data for the corresponding sample eyes, and the maximum mean difference MMD loss between the first domain feature data and the second domain feature data;

[0038] The sample feature data are respectively input into the initial regression decoder for the refractive error recognition task, the initial classification decoder for the cataract recognition task, and the initial segmentation decoder for the pupil segmentation task for processing to obtain the sample reference data for the corresponding task.

[0039] Obtain the task loss between different sample reference data and the corresponding task labels of the input image annotation;

[0040] Based on the weighted total loss of the MMD loss and the losses of the multiple tasks, the network parameters of the initial shared encoder, the initial regression decoder, the initial classification decoder and the initial segmentation decoder are adjusted so as to learn and train the shared encoder with the adjusted network parameters and the decoders of the multiple tasks using images in the dataset.

[0041] The weighted total loss is determined to satisfy the training constraints. Using the finally trained shared encoder and the regression decoder, classification decoder and segmentation decoder for different tasks, a multi-task image recognition model is obtained.

[0042] Optionally, the output of task reference data for each of the plurality of tasks includes at least one of the following implementation methods:

[0043] The eye image to be identified is sent to a display for display, and the task reference data of each of the multiple tasks is displayed on the eye image to be identified;

[0044] According to the preset output format, generate an eye detection report containing task reference data for each of the multiple tasks, and output the eye detection report;

[0045] According to the task identification rules of the different fields, the task reference data of each of the multiple tasks are processed, and the corresponding task identification results are output.

[0046] On the other hand, this application also proposes an eye image recognition device based on multi-task learning, the device comprising:

[0047] An eye image acquisition module is used to acquire an image of the eye to be identified.

[0048] The task reference data acquisition module is used to process the eye image to be identified based on the trained multi-task image recognition model to obtain task reference data for the eye to be identified for multiple tasks in different domains; wherein, the multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different domains, the datasets in different domains include different eye images with corresponding task labels, and the multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task.

[0049] The task reference data output module is used to output the task reference data of each of the multiple tasks, in order to help determine the task recognition results of the target object's eye in different fields.

[0050] In another aspect, this application also proposes a computer device, said computer device comprising:

[0051] Communication interface;

[0052] The memory is used to store programs that implement the eye image recognition method based on multi-task learning as described above;

[0053] A processor is configured to load and execute the program stored in the memory to implement the eye image recognition method based on multi-task learning as described above.

[0054] As can be seen, this application proposes an eye image recognition method and related equipment based on multi-task learning. In order to achieve cataract screening while performing refractive error screening, without increasing additional investment or affecting the refractive error screening process, it proposes to perform multi-task learning based on datasets from different fields to obtain a multi-task image recognition model applicable to three tasks: refractive error recognition, pupil segmentation, and cataract recognition. Compared with the single-task image recognition model, it improves image recognition performance. Thus, after obtaining the eye image to be identified, this application can directly process the eye image to be identified based on this multi-task image recognition model, outputting the task reference data of the eye to be identified in each of the three tasks, effectively assisting doctors to quickly and more accurately determine the task recognition results of the eye to be identified in different fields. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a flowchart illustrating an optional example of the eye image recognition method based on multi-task learning proposed in this application.

[0057] Figure 2 This is a flowchart illustrating another alternative example of the eye image recognition method based on multi-task learning proposed in this application.

[0058] Figure 3 This is a schematic diagram of the system architecture of the eye image recognition method based on multi-task learning proposed in this application;

[0059] Figure 4 This is a flowchart illustrating another possible example of the eye image recognition method based on multi-task learning proposed in this application.

[0060] Figure 5 This is a schematic diagram of the network structure of the multi-task image recognition model proposed in this application;

[0061] Figure 6 This is a schematic diagram of the network structure of the image recognition method for the pupil segmentation task in the eye image recognition method based on multi-task learning proposed in this application;

[0062] Figure 7 This is a flowchart illustrating another possible example of the eye image recognition method based on multi-task learning proposed in this application.

[0063] Figure 8 This is a schematic diagram illustrating the training process of the multi-task image recognition model proposed in this application.

[0064] Figure 9 This is a schematic diagram of an optional example of the eye image recognition device based on multi-task learning proposed in this application.

[0065] Figure 10 A schematic diagram of the hardware structure of an optional example of a computer device for use with the multi-task learning-based eye image recognition method proposed in this application;

[0066] Figure 11 This is a schematic diagram of the hardware structure of another alternative example of a computer device suitable for the multi-task learning-based eye image recognition method proposed in this application. Detailed Implementation

[0067] Regarding the background technology description, with the development and application of artificial intelligence technology, this application proposes to utilize one or more artificial intelligence algorithms to extract features from eye images acquired by eccentric photorefractive equipment for a target task (image data recognition task to assist doctors in screening any ophthalmic disease such as cataracts, glaucoma, diabetes, and refractive myopia; this application uses refractive power recognition task, pupil segmentation task, and cataract recognition task as examples for illustration). Based on this feature extraction, task reference data for the target task can be obtained to assist doctors in quickly determining the patient's condition. Compared to the method of doctors analyzing patient eye images based on clinical experience to obtain corresponding task reference data, this eye image recognition method based on artificial intelligence algorithms improves recognition efficiency and accuracy, reduces the occurrence of misdiagnosis caused by human factors, and can reliably and effectively assist doctors in determining the patient's condition.

[0068] However, in the eye image recognition process described above, the corresponding image recognition model is trained for a single task (such as refractive error recognition, pupil segmentation, or cataract recognition). Since the training process does not consider the hidden relationships between this task and other tasks, the feature data extracted for a single task is often inaccurate and incomplete. This will affect the recognition accuracy of the image recognition model trained for that single task, making it impossible to reliably and effectively assist doctors in recognizing patients' conditions, and may even cause adverse interference to doctors with limited clinical experience.

[0069] To address the aforementioned issues, this application proposes a multi-task learning architecture to train a multi-task image recognition network. This integrates screening tasks from another domain without increasing hardware investment or examination steps, and without affecting the existing refractive screening process (i.e., one domain), thus meeting the multi-task screening needs of different domains. Research shows that, due to the different noise patterns of different tasks, a more general feature representation can be obtained through cross-domain multi-task learning. Furthermore, there are close hidden representations between multiple tasks, including feature representations not present in a single task.

[0070] Therefore, by employing a cross-domain, multi-task learning approach, eye image samples from different domains are trained and learned. More accurate feature data that meets the requirements of different tasks is extracted, resulting in a multi-task image recognition model with higher performance and generalization ability. In practical applications, this multi-task image recognition model is used to extract features from the actual acquired eye images to be recognized, obtaining shared feature data that meets the recognition requirements of multiple tasks. Then, according to the processing needs of different tasks, this shared feature data is analyzed to accurately obtain the task reference data for each task.

[0071] For example, when a multi-task image recognition model is applicable to multiple tasks including refractive error recognition, pupil segmentation, and cataract recognition, the task reference data directly output by the model for each task can be: refractive data of the eye to be identified (such as spherical and cylindrical power), lens opacity (i.e., cataract grading results) and / or cataract category information such as predicted probability / score of cataract category, pupil segmentation results (such as pupil area masks to meet the needs of obtaining various pupil positioning information, or to directly identify the corresponding pupil positioning information), etc. Then, these task reference data can be output to assist doctors in quickly and accurately screening for various ophthalmic diseases and accurately assessing the actual condition of patients.

[0072] As can be seen, this application integrates cataract screening into refractive screening without increasing hardware investment or examination steps, and without affecting the normal refractive screening process. Compared to methods that train corresponding image recognition models for each ophthalmic disease screening task, process the collected eye images, and output reference data for that task, this application's multi-task learning-based eye image recognition method, through shared feature data between multiple tasks, can simultaneously obtain task reference data for multiple tasks. This improves the efficiency, accuracy, and diversity of vision screening, better assisting doctors in identifying various ophthalmic diseases in patients. It should be understood that when task reference data for assisting in the screening of other types of ophthalmic diseases is needed, the corresponding multi-task image recognition model can be obtained using the multi-task learning-based eye image recognition method proposed in this application. The implementation process is similar, and this application will not provide detailed examples.

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. For ease of description, only the parts related to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other. That is, all other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0074] In this invention, flowcharts are used to illustrate the operations performed by the system according to embodiments of the invention. It should be understood that the preceding or following operations are not necessarily performed precisely in sequence. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.

[0075] Based on the above description of the technical concept of the eye image recognition method based on multi-task learning proposed in this invention, the following will use the application scenario of eye diseases such as myopia, cataracts, diabetic retinopathy, and orbital tumors caused by refractive system dysfunction such as refractive errors and lens opacity in patients' eyes as an example to explain in detail the eye image recognition method based on multi-task learning proposed in this invention, which includes but is not limited to the implementation methods described in the embodiments below.

[0076] Reference Figure 1This is a flowchart illustrating an optional example of the eye image recognition method based on multi-task learning proposed in this application. The method can be applied to computer devices, which can be servers and / or terminal devices with certain data processing capabilities. The server can be a standalone physical server, a server cluster integrating multiple physical servers, or a cloud server with cloud computing capabilities, etc. The terminal device can include, but is not limited to, smart medical devices, robots, desktop computers, etc. The terminal device can interact with the server to implement the eye image recognition method based on multi-task learning proposed in this application, and its specific characteristics can be determined according to the requirements of the scenario.

[0077] like Figure 1 As shown, the eye image recognition method based on multi-task learning proposed in this embodiment may include, but is not limited to, the following steps:

[0078] Step S11: Obtain the image of the eye to be identified;

[0079] In practical applications, an eccentric photorefractive device (such as an infrared eccentric photorefractive instrument) can acquire an image of the eye to be identified (such as an infrared pupil image), which is then recorded as the image of the eye to be identified. Following the multi-task learning-based eye image recognition method proposed in this application, task reference data for multiple tasks can be quickly and accurately obtained from this image of the eye to be identified, assisting doctors in screening whether the eye to be identified has refractive errors, cataracts, or other eye diseases. This application does not detail the image acquisition process of the eccentric photorefractive device.

[0080] In cases where the medical imaging equipment, such as an eccentric radiography optometry device, is replaced by the aforementioned computer device, the eye image recognition method based on multi-task learning proposed in this application can be directly executed on the acquired eye image to be recognized. Optionally, if the medical imaging device is not configured with instruction code implementing the eye image recognition method based on multi-task learning proposed in this application, after the medical imaging device acquires the eye image to be recognized, it can transmit the eye image to be recognized to the computer device via a wired or wireless communication network. In this case, the computer device can obtain the eye image to be recognized from the medical imaging device and execute the eye image recognition method based on multi-task learning proposed in this application. This application does not limit the implementation method of step S11.

[0081] Step S12: Based on the trained multi-task image recognition model, the eye image to be recognized is processed to obtain the task reference data of the eye to be recognized for multiple tasks in different fields. The multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different fields. The datasets in different fields include different eye images with corresponding task labels. The multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task.

[0082] Based on the above description of the technical solution of this application, in the process of analyzing the eye image of the eye to be identified, in order to simultaneously predict the task reference data of multiple tasks related to different domains, that is, in the process of refractive examination, i.e., in obtaining the refractive data in the eye image to be identified, it is possible to simultaneously obtain cataract category information such as the degree of lens opacity and / or the predicted probability / score of belonging to the cataract category, to assist doctors in screening whether the eye to be identified has cataracts, a multi-task image recognition model suitable for multi-task image recognition needs can be obtained by first training the eye images of various sample eyes based on a cross-domain multi-task learning method. This multi-task image recognition model can then be used to automatically identify the task reference data for refractive screening and cataract screening. The training process of this multi-task image recognition model can be referred to, but is not limited to, the description in the corresponding part of the following embodiments, which will not be detailed here.

[0083] In the training process of the multi-task image recognition model, the types of multiple tasks and their respective domains, i.e., screening domains, can be determined according to the actual vision screening needs. This application takes the refractive screening domain and the cataract screening domain, as well as the corresponding refractive power recognition task, pupil segmentation task, and cataract recognition task as examples for illustration. The model training methods for other screening domains and their required image recognition task types are similar, and this application will not provide detailed examples.

[0084] Based on this, this application can input the obtained eye image to be identified into a pre-trained multi-task image recognition model for processing. It utilizes the hard-shared characteristics learned between the refractive error recognition task, pupil segmentation task, and cataract recognition task for feature extraction, obtaining image data that simultaneously meets the requirements of refractive error screening and cataract screening—i.e., shared feature data. Then, according to the actual recognition requirements of different tasks, this shared feature data is analyzed separately to obtain corresponding multiple task reference data. Compared to calling a pre-trained image recognition model for each task to process the eye image to be identified and obtain task reference data that meets the recognition requirements of that task, this method improves image recognition efficiency. Furthermore, compared to single-task networks, the performance of multi-task learning networks is significantly improved. Networks trained through cross-domain multi-task learning have higher accuracy and sensitivity, greatly improving the recognition accuracy of the eye image to be identified.

[0085] It should be noted that the above-mentioned multi-task image recognition model can be trained using training datasets from different fields, validated using validation datasets from different fields, and tested using test datasets. The optimized multi-task image recognition model ensures the reliability and accuracy of the processing results for the actual acquired eye images to be recognized.

[0086] Step S13: Output task reference data for each of the multiple tasks to help determine the task recognition results of the eye to be recognized in different fields.

[0087] For a multi-task image recognition model that processes an eye image to be recognized, the task reference data for each of the multiple tasks can be directly displayed on the monitor of a computer device. For example, multiple task reference data can be displayed on the eye image to be recognized output on the monitor, or multiple task reference data can be output in a display area different from the display area where the eye image to be recognized is located, or multiple task reference data can be displayed separately. This application does not limit the implementation method of outputting multiple task reference data on the monitor.

[0088] Optionally, for the output results of the multi-task image recognition model, an eye detection report containing multiple task reference data of the eye to be identified can be generated according to a preset output format, and then the eye detection report can be output, such as by displaying it on a monitor, by outputting it through a voice playback device configured or connected to the computer device, or by sending it to a connected printing device for printing out the eye detection report for the convenience of patients or other doctors to view. This application does not limit the output method of multiple task reference data and can determine it according to actual needs.

[0089] In some embodiments, the task reference data for the refractive error recognition task, pupil segmentation task, and cataract recognition task obtained simultaneously by the multi-task image recognition model can be output according to, but not limited to, the methods described above. Alternatively, at least one task reference data can be processed according to the screening rules of the corresponding refractive error screening field and cataract screening field to obtain the task recognition results of the corresponding field and output them to assist doctors in quickly and accurately determining the visual acuity of the eye to be identified.

[0090] Reference Figure 2 This is a flowchart illustrating another optional example of the eye image recognition method based on multi-task learning proposed in this application. This embodiment can describe an optional refined implementation of the eye image recognition method based on multi-task learning proposed above, such as... Figure 2 As shown, the method may include, but is not limited to:

[0091] Step S21: Obtain the image of the eye to be identified;

[0092] The implementation process of step S21 can be referred to the description in the corresponding part of the context, and will not be described in detail here.

[0093] Step S22: Input the eye image to be identified into the shared encoder in the trained multi-task image recognition model for encoding processing to obtain shared feature data for multiple tasks in different domains; the shared feature data can characterize the hidden relationships between these multiple tasks.

[0094] Based on the contextual description of the process of obtaining a multi-task image recognition model, in the cross-domain multi-task learning process, the learned encoder mainly relies on hard parameter sharing, denoted as the shared encoder, as described above. Figure 3 The diagram shows the system architecture of the eye image recognition method based on multi-task learning. The hard parameter sharing uses the entire backbone network as a shared network to encode information from multiple tasks, that is, to identify some information shared between multiple tasks, share this information, and put multiple related tasks together for learning, so that the learned network has better performance and generalization ability.

[0095] The shared encoder can be constructed using a convolutional neural network. This embodiment does not detail its network structure, but it can be determined based on the feature extraction requirements of multiple tasks in different domains (such as domain features and image features). In practical applications, to extract richer and more accurate features, the shared encoder can include convolutional layers of different scales. By using convolutional kernels of different sizes and with different strides, the received feature maps are convolved to extract receptive field information at different scales. This information is then upsampled to obtain richer shared feature data. The implementation process is not detailed in this application.

[0096] It should be noted that, since the shared encoder learns by using datasets from different domains for multi-task learning, it not only considers the feature representation of each task, but also the close hidden relationships between multiple related tasks, as well as the feature representations that a single task does not have. This is to ensure that the shared feature data after encoding can include richer domain features and image features, and can meet the recognition needs of multiple tasks in different domains.

[0097] Step S23: Input the shared feature data into the decoders of each of the multiple tasks in the multi-task image recognition model for processing to obtain the task reference data of the eye to be recognized in different fields.

[0098] Following the description of the multi-task image recognition model above, in order to obtain the task parameter data of multiple tasks simultaneously using the same input image (i.e., the eye image to be recognized mentioned above), a decoder is configured for each task when constructing the multi-task image recognition model. This decoder learns the mapping from the shared feature data obtained from the shared encoder to the model output, and predicts the task reference data of different tasks for the same eye to be recognized, such as the spherical and cylindrical power of the eye to be recognized, the pupillary region mask, pupillary positioning information, and the degree of lens opacity / cataract type information.

[0099] The network structures of the decoders for the various tasks listed above can be different, and the network structure of each decoder is determined according to the corresponding task type. Optionally, such as... Figure 3 As shown, the decoder for the refractive power recognition task can be constructed using a regression network, and can be referred to as a regression decoder; the decoder for the pupil segmentation task can be constructed using a segmentation network, and can be referred to as a segmentation decoder; the decoder for the cataract recognition task can be constructed using a classification network, such as a binary classifier, and can be referred to as a classification decoder. The network structure of each decoder can be referred to, but is not limited to, the description of the corresponding part of the following embodiment. This embodiment will not be described in detail here.

[0100] It should be noted that since the shared encoder and the decoders corresponding to each of the multiple tasks are obtained through cross-domain multi-task learning, during the training process, after obtaining the output loss of each network, they are weighted and summed. Based on the total loss, the parameters of each network (i.e., the shared encoder and multiple decoders) are adjusted. Compared with the training method of single-task networks, the network based on cross-domain multi-task learning in this application has higher performance, higher accuracy and sensitivity, thereby improving the accuracy of reference data for multiple tasks and effectively assisting doctors in obtaining task recognition results in different domains.

[0101] Step S24: Generate an eye detection report containing task reference data for each of the multiple tasks according to the preset output format, and output the eye detection report.

[0102] For the multiple task reference data obtained by the multi-task image recognition model in processing the eye image to be recognized, the corresponding output method can be determined according to the needs of doctors and patients, and the multiple task reference data can be output accordingly, including but not limited to the output implementation method described in the relevant part of this application.

[0103] In the task reference data output method described in step S24, the preset output format can be a report format determined by vision screening institutions (such as hospitals or communities), medical testing equipment manufacturers, or screening personnel. It can typically record the display methods of different types of task reference data, the display positions in the entire display area (i.e., the display layout between different task reference data), etc. The computer device can process the obtained multiple task reference data according to the preset output format to obtain an eye detection report of the eye to be identified, which can be displayed on the computer device's own monitor or sent to other devices (such as pre-bound terminal devices) for display or printing. This application does not limit the output method of the eye detection report and can be determined as appropriate.

[0104] Optionally, during the aforementioned eye examination report generation process, multiple task reference data can be processed according to task recognition rules for different domains to obtain corresponding task recognition results. For example, the obtained spherical and cylindrical power can be compared with the corresponding refractive thresholds used to distinguish between myopia and hyperopia to determine whether the eye to be identified belongs to the myopia or hyperopia category. Furthermore, refractive recognition results such as the refractive power of myopia or hyperopia can be further determined.

[0105] For the task reference data on lens opacity, the degree of lens opacity of the eye to be identified can be compared with the range of lens opacity corresponding to each lens opacity level (i.e., cataract level) according to preset grading rules to determine the cataract level of the eye to be identified, thus assisting doctors in determining the treatment method for the eye to be identified. Similarly, if the task reference data is cataract category information, such as the predicted probability / predicted score of the eye to be identified having cataracts, it can be used, or the comparison result with the corresponding cataract classification threshold, to determine the task identification result and add it to the identification result display area of ​​the corresponding task reference data in the eye examination report for doctors and patients to view.

[0106] Furthermore, the reference data for the pupil region mask can be analyzed to obtain pupil positioning information such as interpupillary distance, pupil size, and pupil constriction curve. In this way, the output eye examination report can not only display the pupil region mask of the eye to be identified, but also the corresponding pupil positioning information. This can assist doctors in performing vision screening of the eye to be identified, improving the reliability and accuracy of the screening results, and can also help doctors determine cataract treatment plans, etc.

[0107] Therefore, the content of the aforementioned eye examination report can be determined based on the recognition requirements of multiple tasks, including but not limited to the content and output method described above. Furthermore, the preset output format can be adaptively adjusted according to changes in actual needs to update the obtained eye examination report. The implementation process will not be detailed here.

[0108] Reference Figure 4 This is a flowchart illustrating another optional example of the eye image recognition method based on multi-task learning proposed in this application. This embodiment can describe another optional refined implementation of the eye image recognition method based on multi-task learning proposed above, such as... Figure 4 As shown, the method may include, but is not limited to, the following steps:

[0109] Step S41: Obtain the eye image to be identified;

[0110] The implementation process of step S41 can be referred to the description in the corresponding part of the context, and will not be described in detail here.

[0111] Step S42: Input the eye image to be identified into the shared encoder in the trained multi-task image recognition model, extract features through the first scale convolutional layer, and obtain the first feature map after processing by the max pooling layer.

[0112] Step S43: Input the first feature map into the second-scale convolutional layer and the first-scale convolutional layer in sequence for feature extraction. After processing by the max pooling layer, the second feature map is obtained.

[0113] Step S44: The second feature map is sequentially input into two first-scale convolutional layers for feature extraction. After processing the obtained third feature map by the max pooling layer, shared feature data for multiple tasks is obtained.

[0114] Based on the above description of multi-task image recognition models, and referring to... Figure 5The diagram shows the network structure of a multi-task image recognition model. The shared encoder can be composed of a multi-scale convolutional neural network. The scale of the convolutional kernels in each convolutional layer of this convolutional neural network can be determined based on the size of the eye image to be recognized (which can be a grayscale image). This application does not limit the scale of each convolutional kernel or the stride of the convolution operation. Specifically, in the different convolutional layers described above, the first scale of the convolutional kernels is smaller than the second scale. The first-scale convolutional kernel performs the convolution operation according to the first step length (such as kernel sliding), and the second-scale convolutional kernel can perform the convolution operation according to the second stride length, where the first step length is smaller than the second stride length. This application does not limit the values ​​of the hyperparameters of each network layer and can determine them as needed.

[0115] In some embodiments, such as Figure 5 As shown, the shared encoder can be constructed sequentially from the following layers: a convolutional layer with a kernel size of 3×3 (denoted as the first scale) and a stride of 1; a MaxPool pooling layer; a convolutional layer with a kernel size of 11×11 (denoted as the second scale) and a stride of 2; a convolutional layer with a kernel size of 3×3 and a stride of 1; a MaxPool pooling layer; two convolutional layers with a kernel size of 3×3 and a stride of 1; and a MaxPool pooling layer. Figure 5 The hyperparameters of each network layer can be represented as input channels, kernel size, output channels, stride, and padding, but are not limited to these hyperparameter types and their representations.

[0116] Based on this, after the eye image to be identified is input into the shared encoder, a 3×3 convolutional layer with a stride of 1 directly performs a convolution operation on the eye image, i.e., a feature extraction operation. The obtained feature data is then input into a MaxPool pooling layer for processing to reduce computation and prevent overfitting, resulting in the first feature map. This first feature map is then sequentially input into an 11×11 convolutional layer with a stride of 2 for convolution. The resulting feature map is then input into a 3×3 convolutional layer with a stride of 1 for further convolution, achieving feature extraction at different scales. After processing by a max pooling layer, a second feature map with rich feature data at different levels is obtained. This second feature map is then sequentially input into two 3×3 convolutional layers with a stride of 1 for convolution. After processing by the max pooling layer, the extracted feature data is obtained, resulting in shared feature data. The processing procedures of the input data by each network layer are not detailed in this embodiment.

[0117] As can be seen, compared with the feature data obtained by the encoder of a single-task network, the shared feature data obtained by the above method in this application contains richer domain features and image features, which better meets the image recognition needs of the single task, and can also meet the image recognition needs of other related tasks, thus improving the efficiency and accuracy of multi-task image recognition of the same input image.

[0118] It should be noted that before inputting the trained shared encoder into the eye image to be recognized, it is possible to first determine whether the image size of the eye image to be recognized meets the input image size requirements of the shared encoder. If it does not meet the requirements, the eye image to be recognized can be processed according to the input image size requirements. Then, the processed eye image to be recognized that meets the input image size requirements can be input into the shared encoder, and multi-scale feature extraction can be performed according to the method described above to obtain the shared feature data required for multiple preset tasks, i.e., the shared feature map.

[0119] Step S45: Input the shared feature data into the regression decoder for the refractive error recognition task, and obtain the refractive reference data of the eye to be identified through regression processing of two fully connected layers.

[0120] In practical applications, in order to accurately screen whether the eye to be identified has refractive errors, the multi-task image recognition model constructed in this application includes a regression decoder for the refractive error recognition task. The shared feature data is processed to obtain the refractive data in the image of the eye to be identified, which serves as the refractive reference data of the eye to be identified, such as the values ​​of spherical power and cylindrical power that characterize the refractive error of the eye to be identified. This helps doctors determine whether the eye to be identified is myopic or hyperopic, that is, to accurately screen whether the eye to be identified has refractive errors.

[0121] Step S46: Input the shared feature data into the classification decoder for the cataract recognition task. After classification processing through two fully connected layers, normalize the classification results to obtain the predicted probability that the eye to be identified belongs to the cataract category.

[0122] Since cataracts are an ophthalmic disease caused by clouding of the refractive medium (lens), which is located in the center of the eye's light path, clouding directly affects a patient's daily life. Therefore, this application proposes adding cataract screening to refractive examinations to detect cataract patients early and reduce the possibility of cataract-induced blindness. To this end, following the technical solution described above, this application will, without increasing hardware investment or examination steps and without affecting the normal refractive screening process, employ cross-domain multi-task learning technology to obtain a multi-task image recognition model for both refractive and cataract screening. Specifically, a classification decoder for the cataract recognition task will be added to its decoding network. The model structure can be referred to the description in the relevant sections of the context; this embodiment will not elaborate further.

[0123] Based on this, after obtaining the shared feature data according to the method described above, during the refractive error recognition process, the shared feature data can also be input into the classification decoder. In this classification decoder, such as... Figure 5As shown, the input feature data can be classified by two sequentially connected fully connected layers. After normalization by a Softmax activation function, the probability distribution of the data on the two labels of cataract and non-cataract categories is obtained. The predicted probability that the eye to be identified belongs to the cataract category is determined. The prediction probability is then compared with a preset cataract threshold (which can be a critical probability for distinguishing whether an eye image is an image of a cataract patient's eye, determined based on experience or a large number of experiments; this application does not limit its value) to determine whether the eye to be identified has cataracts.

[0124] It should be noted that the output of the above classification decoder includes, but is not limited to, the predicted probability described in step S46. It can also be the predicted score obtained therefrom, or the lens opacity data of the eye to be identified (such as the lens opacity degree mentioned above). This lens opacity data can characterize the cataract level of the eye to be identified. Based on this, the classification decoder can also output cataract category information such as the predicted probability that the eye to be identified belongs to the cataract category and the lens opacity data. This application does not limit the output content of the classification decoder. The task label content of the cataract identification task during model training can be adjusted according to actual needs so that the classification decoder can output corresponding content. The implementation process is not described in detail in this embodiment.

[0125] Step S47: Input the shared feature data into the segmentation decoder for the pupil segmentation task. After feature extraction through convolutional layers of different scales, the feature maps output by the connected convolutional layers are processed through multiple upsampling layers or feature joint layers. The obtained feature maps are then input into the unit convolutional layer for feature data processing to obtain the pupil region mask of the eye to be identified.

[0126] Following the above analysis, in order to accurately obtain pupil localization information such as pupil size, pupil position, interpupillary distance, gaze direction, and pupil constriction curve during refractive screening, it is also necessary to obtain a pupil region mask of the eye image to be identified, i.e., a mask image of the pupil region. Therefore, when constructing the decoding network of the multi-task image recognition model, a segmentation decoder for the pupil segmentation task can also be configured. This segmentation decoder can be constructed using, but is not limited to, deep image semantic segmentation networks (such as the SegNet network structure).

[0127] Based on this, refer to Figure 6 The diagram shown illustrates the network structure of an image recognition method for pupil segmentation. Based on the shared encoder network structure described above, a segmentation decoder can be constructed using the SegNet algorithm, and the parameters of each network layer in this segmentation decoder can be determined. For example... Figure 6As shown, since the processing of input data by certain network layers in the segmentation decoder needs to be based on the processing results of the corresponding convolutional layers in the shared encoder, in order to supplement the low-order feature data lost during the convolution operation performed by the corresponding convolutional layers, it can be seen that after the network structure of the shared encoder changes, the network structure of the segmentation decoder can be adjusted accordingly to ensure the accuracy of the output results. This adjustment process can be obtained through the synchronous learning of the shared encoder and each decoder during the model training process, which will not be described in detail in this embodiment.

[0128] Therefore, the convolutional layers of different scales in the above segmentation decoder may include multiple first-scale convolutional layers and one second-scale convolutional layer; the feature fusion layer is connected to the corresponding convolutional layer of the shared encoder to realize the feature fusion operation between the feature map output by the convolutional layer and the feature map output by the convolutional layer connected in the segmentation decoder, so as to increase the low-order feature data lost by the feature extraction process of the convolutional layer connected in the segmentation decoder.

[0129] For example, in combination Figure 5 The diagram illustrates the parameters of each network layer in the shared encoder. For the shared feature data input to the segmentation decoder, based on the network layers constituting the decoder, features are first extracted from the shared feature data using a 3×3 convolutional layer with a stride of 1 to enrich the feature data. Then, an upsampling layer (which can be an upsampling layer with a ReLU activation function; the type of activation function can be determined based on the activation function used in the pooling layers between different convolutional layers in the shared encoder, and is not limited to this one activation function; it can be adaptively adjusted according to actual needs) is used to restore the original image size, ensuring that the size of the processed feature map meets the input image size requirements of the next convolutional layer. Figure 5 As shown, features can still be extracted from the input feature map using a convolutional layer with a kernel size of 3×3 and a stride of 1. The implementation process will not be described in detail in this embodiment.

[0130] To compensate for the low-order feature data lost by each convolutional layer in the encoding process of the eye image to be recognized during the shared encoder, the fourth network layer of the segmentation decoder can adopt a feature concatenation layer. This layer can obtain the feature maps output by the previous convolutional layer and the corresponding convolutional layer in the shared encoder, and perform a feature concatenation operation on the feature data in the obtained feature maps.

[0131] Then, the processing can be repeated four times following the sequential steps of convolutional layer, upsampling layer, convolutional layer, and feature union layer described above, to repeatedly restore the image feature data and ensure the accuracy of the output results. It should be noted that, in order to obtain a larger field of view feature data during this process, the kernel size of the second convolutional layer can be 11×11 and the stride of the convolution operation can be 2 when the process is repeated for the third time, but it is not limited to these.

[0132] Furthermore, the feature data output from the last feature joint layer, after feature extraction by a 3×3 convolutional layer with a stride of 1, can be further extracted using a unit convolutional layer with a sigmoid activation function. This reduces the number of output channels and yields a mask for the pupil region of the eye to be identified. The processing of input data by each network layer in the segmentation decoder can be implemented based on the type of the corresponding network layer and its network parameters; this embodiment will not elaborate on these details.

[0133] Step S48: Process the pupil region mask to obtain the pupil positioning information of the eye to be identified;

[0134] Based on the actual required pupil positioning information, this application can use corresponding calculation methods to process the pupil region mask, such as calculating the pupil distance, identifying the pupil size, and constructing the pupil contraction curve by using the pupil size of the pupil region mask of multiple consecutive frames of eye images to be identified. This application does not limit the implementation process of step S48 and can be determined as appropriate.

[0135] Step S49: Output the refractive reference data of the eye to be identified, the predicted probability of belonging to the cataract category, the pupil area mask, and the pupil positioning information.

[0136] Regarding the output method of step S49, it can be implemented with reference to the description in the corresponding part of the context, and will not be described in detail here. In addition, for different task reference data of the eye to be identified obtained by the same multi-task image recognition model, different output methods can also be used for output, which can be determined according to actual needs.

[0137] Reference Figure 7 This is a flowchart illustrating another optional example of the eye image recognition method based on multi-task learning proposed in this application. This embodiment can describe the training and learning process of the multi-task image recognition model involved in the context, such as... Figure 7 As shown, the method may include:

[0138] Step S71: Obtain raw eye images of multiple sample eyes;

[0139] Based on the above description of the multi-task image recognition model, this application can be implemented using cross-domain multi-task learning technology. In the process of obtaining datasets from different domains, raw eye images of different sample eyes, such as infrared pupil images, can be collected first through medical testing equipment such as eccentric photorefractive equipment. Then, the corresponding task labels for the refractive screening domain and cataract screening domain are obtained, resulting in eye images with task labels from different domains.

[0140] In order to improve the reliability and accuracy of model training, the above-mentioned multiple sample eyes can be different types of eyes, such as the eyes of patients with at least one ophthalmic disease such as myopia, hyperopia or cataracts, the eyes of patients can be the eyes of patients with different degrees of ophthalmic diseases, and can also include the eyes of patients without ophthalmic diseases, etc. This application does not limit the types of multiple sample eyes.

[0141] Step S72: Analyze the original eye image to obtain the refractive data of the corresponding sample eye, and determine the refractive data as the refractive power task label of the original eye image;

[0142] The refractive data may include refractive power values ​​such as spherical and cylindrical power. This application does not restrict the calculation method for obtaining refractive data from the original eye image; it can be determined based on the calculation principle of the corresponding refractive data. To implement a supervised training method, this application can define the refractive data as the refractive power task label of the original eye image of the corresponding sample eye.

[0143] Step S73: Perform segmentation processing on the original eye image to obtain the pupil mask image and pupil positioning information of the corresponding sample eye, and determine the pupil mask and pupil positioning information as the segmentation task label of the original eye image.

[0144] In this embodiment, a segmentation algorithm can be used to segment the original eye image to determine the pupil region mask image, i.e., the pupil mask image. If necessary, the pupil mask image can be further processed to obtain the pupil positioning information as described above, so as to determine the pupil mask image and the pupil positioning information as the segmentation task label of the corresponding original eye image.

[0145] Optionally, after obtaining the refractive data and pupil localization information of the same original eye image of the same sample eye, this application can add them to the original eye image or its pupil mask image, and determine the image with the corresponding segmentation task label as the sample image for subsequent model training.

[0146] Step S74: Obtain the cataract category information of each of the multiple sample eyes, and determine the cataract category information as the cataract task label of the corresponding original eye image;

[0147] Based on the above description of the technical solution of this application, this application aims to achieve cataract screening while assisting in refractive examination. Therefore, in the process of acquiring the dataset for model training (which may also include model validation and model testing as needed), according to the method described above, while obtaining the task labels for the refractive recognition task and pupil segmentation task in the field of refractive screening from the original eye images, the cataract task label for the cataract recognition task in the field of cataract screening can also be obtained.

[0148] Therefore, for the original eye images of the above-mentioned multiple sample eyes, doctors can use professional techniques to determine whether they are images of cataract patients. For example, a slit-lamp microscope can be used to view the corresponding sample eye, and the image data of the sample eye acquired by the slit-lamp microscope can be analyzed to determine whether the sample eye is the eye of a cataract patient. The cataract category information obtained is then input into a computer device and identified as the cataract task label for each original eye image of the corresponding sample eye. This application does not restrict the input method of cataract category information and it can be determined as appropriate.

[0149] In some other embodiments, during the process of obtaining the above-mentioned cataract category information, the lens opacity data of the corresponding sample eye can also be obtained from the eye images collected by medical testing equipment such as slit-lamp microscopes or eccentric radiography equipment. If determined by the observation results of the corresponding sample eye by a professional doctor through a slit-lamp microscope, the lens opacity data and cataract category information can then be determined as cataract task labels for subsequent model training and learning.

[0150] Step S75: Using the original eye image labeled with refractive power task label, the pupil mask image labeled with segmentation task label and refractive power task label, and the original eye image labeled with cataract task label, datasets from different domains are obtained.

[0151] Following the above analysis, to meet the need for simultaneous screening of the same eye in both the refractive error screening domain (referred to as the first domain for convenience) and the cataract screening domain (referred to as the second domain for convenience), this application will employ cross-domain multi-task learning technology for model training. When preparing the datasets required for model training, datasets for each domain can be obtained separately according to the methods described above. For example, for each original eye image of each sample eye, according to, but not limited to, the methods described above, original eye images labeled with refractive error task tags and pupil mask images labeled with segmentation task tags and refractive error task tags (which can also be original eye images labeled with segmentation task tags, or original eye images labeled with segmentation task tags and refractive error task tags, etc.) are obtained, constituting the dataset for the first domain. Similarly, for each original eye image of each sample eye, according to, but not limited to, the methods described above, original eye images labeled with cataract task tags are obtained, constituting the dataset for the second domain.

[0152] Step S76: Based on a cross-domain multi-task learning approach, an initial image recognition model is constructed. This initial image recognition model includes an initial shared encoder, an initial regression decoder for the refractive error recognition task, an initial classification decoder for the cataract recognition task, and an initial segmentation decoder for the pupil segmentation task.

[0153] For the network structure of the initial image recognition model, please refer to the above text. Figure 5 and Figure 6 The relevant descriptions are not detailed in this embodiment.

[0154] It should be understood that for each image in the datasets of different domains, that is, the images with task labels obtained by the method described above, the image size is consistent with the image size requirement of the input image in the initial image recognition model. Specifically, the input image size requirement of the network layer in the initial image recognition model can be determined according to the image size of each image in the obtained dataset, so that the constructed initial image recognition model can adapt to the size of the input image. The implementation process is not detailed in this application.

[0155] Step S77: Input the images contained in the datasets of different domains into the initial shared encoder for feature extraction to obtain first domain feature data and second domain feature data;

[0156] Reference Figure 8The diagram illustrates the training process of the multi-task image recognition model. After obtaining the datasets for the first and second domains using the method described above, images from different domains can be input into the initial shared encoder for cross-domain feature extraction. The process can be described in conjunction with the above description of the network structure of the shared encoder. After processing through convolutional layers of different scales and max pooling layers deployed between the convolutional layers, feature data (i.e., feature vectors) for the first and second domains can be obtained. The processing steps are not detailed in this embodiment.

[0157] For ease of description, the datasets D obtained above from different domains can be represented as:

[0158]

[0159] in, This can represent the dataset of the k-th domain. Since this application obtains datasets of two domains, k = 2. The m-th task can represent the task label for the m-th task in two domains. In this application, the image recognition tasks can include three categories: refractive error recognition, cataract recognition, and pupil segmentation. The corresponding task labels are obtained according to the method described above, therefore, m = 3. i can represent the number of images contained in the dataset.

[0160] Based on this, in the above formula (1), It can represent a dataset from the first domain. It can represent a dataset from the second domain. A task label can represent the diopter recognition task for the i-th image; This can represent the diopter task label for the i-th image; This can represent the segmentation task label for the i-th image; The cataract task label can be represented by the i-th image. The output of the k-th task (i.e., the task reference data) can be represented as:

[0161]

[0162] The loss function used in the multi-task learning process of the initial image recognition model can be expressed as:

[0163]

[0164] In the above formula (3), ζ can represent an empirical loss. Each task in the model (the task corresponding to each of the three decoders) has its own weight λ to balance the system loss. In the process of cross-domain multi-task learning, the method of minimizing distribution differences can be introduced to integrate the feature differences introduced by different domains. Therefore, based on formula (3), the loss calculation of minimizing distribution differences, that is, the shared encoder output loss calculation, can be added to improve the output accuracy of the trained model.

[0165] Step S78: Minimize the distribution difference processing of the first domain feature data and the second domain feature data to obtain the sample feature data for the corresponding sample eyes, and the maximum mean difference MMD loss between the first domain feature data and the second domain feature data.

[0166] Following the above analysis, to reduce the differences in feature data across different domains, this application employs the MMD (Maximum Mean Discrepancy) method (a loss function used to measure the distance between the distributions of two different but related random variables) to obtain the distance between feature distributions in different domains. By continuously reducing the distance between feature distributions, the feature differences between different domains are reduced, thus obtaining shared feature data for multi-task learning. If one of the two domains is taken as the source domain and the other as the target domain, their respective datasets can be denoted as D. s ={x1,x2,...,x n} and D t ={y1,y2,...,y m}, where n and m are the number of images contained in the corresponding dataset, and this application does not impose any restrictions on their values.

[0167] In order to learn a feature extractor suitable for the above three tasks That is, a shared encoder transforms the input image space x into a reproducible reproducing Hilbert space (RKHS) Н, thereby making the source domain feature distribution D s and target domain feature distribution D t The distance between them is minimized, meaning the distance between the distributions of the feature data in the first domain and the distributions of the feature data in the second domain is minimized. The MMD calculation formula used to measure the distance between the feature distributions of the source and target domains can be:

[0168]

[0169] Due to RKHS's assumptions about the feature space, the MMD statistic helps to balance the trade-off between overfitting and underfitting. MMD(D) can be used to minimize the distributional variance. s D T )2 This is achieved by selecting a suitable kernel function k and minimizing MMD(D). s D T ) 2 The calculation process can be expressed as:

[0170]

[0171] Incorporating domain-adaptive technologies, and in order to learn domain-invariant features, this application can employ a learnable bilinear kernel function *k* to capture the relationship between source and target domain features, i.e. Therefore, the kernel function k in the above formula (5) can be expressed as: k(x,y)=x T (W T Substitute W+I)y+c into formula (5) to perform MMD calculation.

[0172] In the formula for calculating the kernel function k, W can represent a learnable non-zero weight matrix, I can represent the identity matrix, and c can be a learnable scalar. Within the regenerated Hilbert space, (W...) in the kernel function... T W+I) to maintain positive definiteness. Based on this, the loss calculation of the shared encoder, i.e., the maximum mean difference (MMD) loss between the first and second neighborhood feature data, can be obtained using the following constrained MMD loss calculation formula:

[0173] L MMD =MMD(D s D T ) 2 -log(||W||2)-log(c) (6)

[0174] In this embodiment, the first domain feature data and the second domain feature data can be substituted into formulas (5) and (6) to obtain the sample feature data of the corresponding sample eye, i.e., the shared feature vector. As shown in formula (6), in this calculation process, regularization can be used for the above W to avoid W from saturating to zero, while c needs to remain non-negative. The calculation process is not described in detail in this application.

[0175] Step S79: Input the sample feature data into the initial regression decoder for the refractive error recognition task, the initial classification decoder for the cataract recognition task, and the initial segmentation decoder for the pupil segmentation task respectively for processing to obtain the sample reference data for the corresponding task.

[0176] Based on the above descriptions of the network structures of the initial regression decoder, initial classification decoder, and segmentation decoder, it can be understood that for different tasks described above, the constructed corresponding decoders can be used to decode the same sample feature data to obtain the corresponding task reference data, denoted as sample reference data. The decoding process can refer to the above description of the process of obtaining task reference data, which will not be detailed in this embodiment.

[0177] Step S710: Obtain the task loss between different sample reference data and the corresponding task labels annotated on the input image;

[0178] Since the network structure of the decoder is different for different tasks, the processing requirements for the decoding results (such as the sample reference data mentioned above) for different tasks may be different. Therefore, different loss functions can be used to obtain the corresponding task loss for the output results of different decoders.

[0179] Optionally, this application can use the mean squared error (MSE) as the loss function for the refractive error recognition task to obtain the mean squared error loss for the refractive error recognition task. Therefore, the task loss for this refractive error recognition task... The following formula can be used for calculation:

[0180]

[0181] In formula (7), p b It can represent the refractive error recognition task label content of the corresponding input image (such as any image included in the dataset above), that is, the expected output of the corresponding sample eye; It can represent the prediction result output by the regression decoder, that is, the sample reference data of the diopter recognition task obtained by the above prediction, and n can represent the number of images contained in the dataset, that is, the number of samples.

[0182] Optionally, for the task loss calculation of the pupil segmentation task described above, the binary cross-entropy (BCE) function can be used as its loss function. Therefore, the task loss ζ for the pupil segmentation task of the input image is... BCE The formula for calculating (o,s) can be:

[0183] ζ BCE (o,s)=-(olog(s)+(1-o)·(log(1-s))) (8)

[0184] In formula (8), o can represent the segmentation task label, i.e. the real pupil mask image, and s can represent the pupil region mask predicted by the segmentation decoder, i.e. the sample reference data of the pupil segmentation task.

[0185] Furthermore, the task loss for the aforementioned cataract recognition task can be calculated using categorical cross-entropy (CE) as its loss function. Therefore, the task loss for the cataract recognition task with the input image can be calculated accordingly. It can be represented as:

[0186]

[0187] In formula (9), p c This can represent the cataract task label for the corresponding image annotation cataract recognition task (i.e., a binary classification task). The output of the classification decoder can represent the predicted probability / score of whether the corresponding sample eye has cataracts, predicted from the input image, and / or sample reference data such as lens opacity data, which can be determined based on the cataract task label content. N can represent the number of classification categories. In this application, N can be used.

[0188] It should be noted that, depending on actual needs, the loss function used in the loss calculation of the sample reference data output by the decoder for the different tasks described above can be adaptively adjusted, including but not limited to the loss function categories described above.

[0189] Step S711: Based on the weighted total loss of MMD loss and multiple task losses, adjust the network parameters of the initial shared encoder, initial regression decoder, initial classification decoder and initial segmentation decoder respectively, so as to learn and train the shared encoder with adjusted network parameters and the decoders of the multiple tasks respectively through images in the dataset.

[0190] Step S712: Determine that the weighted total loss satisfies the training constraints, and use the finally trained shared encoder and the regression decoder, classification decoder and segmentation decoder for different tasks to obtain a multi-task image recognition model.

[0191] Based on the above description of the network loss function in cross-domain multi-task learning, the total loss function ζ of the cross-domain multi-task learning network can be obtained using the following formula:

[0192] ζ=ω1ζ MSE +ω2ζ CE +ω3ζ BCE +ω4ζ MMD (10)

[0193] In formula (9), ζ MSE The loss function ζ can be used to represent the task loss for obtaining refractive power recognition. CE ζ can represent the loss function used to obtain the task loss for the cataract recognition task. BCE ζ can represent the loss function used to obtain the task loss for pupil segmentation. MMD This can represent the loss used to obtain the output of the shared encoder, i.e., the loss function that minimizes the differences in feature distribution across different domains. ω1, ω2, ω3, and ω4 represent the weights of the corresponding tasks, i.e., optimization constants, which can be optimized during the continuous learning and training of the image recognition model to reduce the total loss.

[0194] After weighted summation of the losses of multiple tasks and the MMD loss according to the above formula (10), the total weighted loss of the image recognition model can be obtained. It can be determined whether the total weighted loss reaches the loss threshold or whether it meets the training constraints. If it does not meet the requirements, the network parameters of the shared encoder and each decoder that constitute the image recognition model can be adjusted according to the total weighted loss. After obtaining the image recognition model with the adjusted network parameters, the new image recognition model can be used to process the images contained in the datasets of different fields according to the multi-task learning method described above, and it can be determined whether the corresponding total weighted loss meets the training constraints.

[0195] After multiple training iterations using the above method, if the weighted total loss obtained in a particular training iteration satisfies the training constraints, a multi-task image recognition model can be constructed using the finally trained shared encoder and regression decoders, classification decoders, and segmentation decoders for different tasks. The training process for each iteration is similar and will not be detailed in this application. Furthermore, the aforementioned training constraints include, but are not limited to, the weighted total loss being less than the loss threshold described above; they can also include the weighted total loss converging, etc., and can be determined based on actual needs.

[0196] In practical applications, after a multi-task image recognition model is trained on a computer device, it can be associated with the task types of multiple tasks or the domain types of two domains and then stored. If the computer device is not a server, the trained multi-task image recognition model can be uploaded to a server for storage; the storage process is not detailed here. Thus, when refractive error screening and cataract screening are needed for a specific eye, after obtaining the image of the eye to be identified, the trained multi-task image recognition model can be retrieved. The image of the eye to be identified can be directly input into the multi-task image recognition model for processing, directly outputting multiple task reference data, such as refractive data, pupil area mask, cataract category information, etc., to assist doctors in determining whether the eye to be identified has refractive errors, cataracts, or other ophthalmic diseases, and further, to determine appropriate treatment methods.

[0197] The multi-task image recognition model trained above can be validated and tested to ensure that the output results of the stored multi-task image recognition model are reliable and accurate. Based on this, datasets from different domains can be obtained according to the method described above, such as eye images containing 890 sample eyes. These eye images can be labeled with task tags for different tasks. Then, the entire dataset can be grouped according to a certain ratio (such as 7:2:1, which can be determined according to actual needs) to obtain a training dataset (which may include eye images of 623 sample eyes), a validation dataset (which may include eye images of 178 sample eyes), and a test dataset (which may include eye images of 89 sample eyes).

[0198] Based on the above descriptions of the dataset acquisition process in different fields, when selecting sample eyes, a certain ratio can be chosen between sample eyes with cloudy lenses and sample eyes with clear lenses. For example, 373 sample eyes with cloudy lenses and 517 sample eyes with clear lenses can be selected. After acquiring the corresponding raw eye images using the above method, multiple raw eye images can be acquired for each sample eye. After filtering out some unqualified images (such as blurry images, images with incomplete pupil areas, etc.), the qualified raw eye images can be processed to obtain images with different task labels to form a dataset. For example, 27,253 raw eye images (such as infrared pupil images) can be obtained to achieve cataract recognition, that is, to identify sample eyes with cloudy lenses.

[0199] Based on this, to verify the performance of the multi-task image recognition model in the refractive error identification task, 10% of the eye images of 89 sample eyes were randomly selected from the total dataset to form a test dataset. With 80 samples correctly classified as spherical power and 73 samples correctly classified as cylindrical power, the multi-task image recognition model was used to obtain the number of different predicted refractive errors for the sample glasses labeled as positive or negative for both spherical and cylindrical power. The test results are shown in Table 1 below. It can be seen that the multi-task image recognition model has high accuracy in predicting whether the sample eyes are myopic or hyperopic, and can effectively assist doctors in determining whether the sample eyes have refractive errors.

[0200] Table 1

[0201] project spherical power of refractive screening Cylindrical power of refractive screening Refractive error prediction 7(7.87％) 0 Refractive negative prediction negative 73(82.01％) 73(82.02％) Refractive positive predicts negative 4(4.49％) 0 Refractive negative prediction positive 5(5.62％) 16(17.98％) Refractive Classification Prediction 80(89.89％) 73(82.02％) Total number of sample eyes 89(100％) 89(100％)

[0202] Table 2

[0203]

[0204] If we take 1.5D as the refractive threshold and statistically analyze the number of differences between the predicted and true values, using the test dataset of the aforementioned 89 sample eyes as an example, we obtain the difference between the predicted value (i.e., the predicted result output by the multi-task image recognition model) and the true value (i.e., the corresponding task label) for different refractive powers (spherical power S, cylindrical power C) of the same sample eye. This difference is then compared with 1.5D, and the number of different comparison results is counted. The test results are shown in Table 2. The accuracy of the predicted value is very high, which can meet the accuracy requirements of refractive screening scenarios.

[0205] Furthermore, following the test dataset example described above, this application can also compile a table of the average difference between predicted and true values, thereby obtaining the mean absolute error S between the predicted and true values ​​for all sample eyes of spherical power S. _ MAE and mean squared error S _ MSE, as shown in Table 3, can also be used to obtain the S corresponding to other items. _ MAE and S _ MSE; similarly, we can also obtain the S corresponding to each of the multiple items of cylinder power C. _ MAE and S _ The implementation process of MSE is not detailed in this application.

[0206] Table 3

[0207]

[0208] As shown in Table 3, the prediction bias of spherical power in the test dataset remains at around 0.5D, and the error is even smaller in sample eyes with positive true values; the prediction bias of cylindrical power also remains at around 0.5D, which is close to the error in spherical power, thus meeting the requirements for refractive power recognition.

[0209] To validate and optimize the performance of the multi-task image recognition model in the cataract recognition task, specifically its classification performance, the classifier can be implemented using the Adam optimizer. The total dataset, consisting of 27,253 eye images from 890 sample eyes, is divided into three groups according to the aforementioned 7:2:1 ratio, resulting in a training dataset containing 19,077 eye images, a validation dataset containing 5,451 eye images, and a test dataset containing 2,725 eye images.

[0210] In practical applications, when determining whether the lens of a sample eye is cloudy, it can be determined by multiple recognition results from multiple eye images of the sample eye. Assuming that more than 50% of the eye images in the sample image show a cloudy lens, the lens of the sample eye can be considered cloudy; conversely, if more than 50% of the eye images show a transparent lens, the lens of the sample eye can be considered transparent. To demonstrate that the classification performance of the multi-task image recognition model described in this application is superior to that of a single-task image recognition model, the multi-task image recognition model proposed in this application can be compared with at least one single-task image recognition model, such as the AlexNet model and the DenseNet-161 model, using the aforementioned validation and test datasets.

[0211] The confusion matrix for predicting and identifying the cataract category in the eye images of each sample in the validation dataset can be:

[0212]

[0213] The confusion matrix for predicting the cataract category in the eye images of each sample in the test dataset can be:

[0214]

[0215] Table 4

[0216] Model type accuracy Specificity Sensitivity accuracy F1 value AlexNet 87.07％ 93.21％ 78.63％ 89.41％ 83.68％ DenseNet-161 86.52％ 93.90％ 77.48％ 91.21％ 83.79％ Multi-task image recognition model 94.39％ 95.90％ 92.53％ 94.85％ 93.68％

[0217] Table 5

[0218] Model type accuracy Specificity Sensitivity accuracy F1 value AlexNet 84.26％ 90.39％ 75.64％ 84.85％ 79.98％ DenseNet-161 83.16％ 90.01％ 74.37％ 85.30％ 79.46％ Multi-task image recognition model 91.01％ 92.03％ 89.71％ 89.78％ 89.74％

[0219] Based on the eye images contained in the aforementioned validation and test datasets, the multi-task image recognition model proposed in this application is compared with the AlexNet and DenseNet-161 models in terms of accuracy, specificity, sensitivity (recall), precision, and F1 score, as shown in Tables 4 and 5 above. It is evident that, on both the validation and test datasets, the multi-task image recognition model proposed in this application significantly outperforms the single-task image recognition models AlexNet and DenseNet-161.

[0220] Similarly, for the performance testing of the multi-task image recognition model on the pupil segmentation task, the test dataset contains various images. These images are sequentially input into the multi-task image recognition model for processing, resulting in pupil region masks for the corresponding images. Combining this with the segmentation task label of the pupil mask image annotated with the input images, the mean intersection-over-union ratio (MIU) of the multi-task image recognition model can be calculated to be 0.8472, and the dice coefficient to be 0.9637. This indicates that the pupil segmentation performance meets the requirements for obtaining pupil localization information.

[0221] In summary, the multi-task image recognition model trained as described above can simultaneously obtain the task reference data required for three types of tasks: refractive error recognition, cataract recognition, and pupil segmentation. Compared with the output results obtained by the image recognition model trained on a single task, the output of the multi-task image recognition model is more reliable and accurate, and can effectively assist doctors in quickly and accurately performing refractive error screening and cataract screening of the eye to be identified.

[0222] Reference Figure 9 This is a schematic diagram of an optional example of the eye image recognition device based on multi-task learning proposed in this application, as shown below. Figure 9 As shown, the device may include

[0223] Eye image acquisition module 91 is used to acquire an eye image of the eye to be identified;

[0224] The task reference data acquisition module 92 is used to process the eye image to be identified based on the trained multi-task image recognition model to obtain task reference data for multiple tasks in different domains of the eye to be identified; wherein, the multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different domains, the datasets in different domains include different eye images with corresponding task labels, and the multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task.

[0225] The task reference data output module 93 is used to output the task reference data of each of the multiple tasks to assist in determining the task recognition results of the target object's eye in different fields.

[0226] In some embodiments, a dataset acquisition module for obtaining datasets from different domains may include:

[0227] The refractive power task label determination unit is used to analyze the original eye image, obtain the refractive data corresponding to the sample eye, and determine the refractive data as the refractive power task label of the original eye image; the refractive data includes spherical power and cylindrical power;

[0228] The segmentation task label determination unit is used to segment the original eye image to obtain the pupil mask image and pupil positioning information corresponding to the sample eye, and to determine the pupil positioning information as the segmentation task label of the original eye image.

[0229] The cataract task label determination unit is used to acquire lens opacity data and / or cataract category information for each of the multiple sample eyes, and to determine the lens opacity data and / or the cataract category information as the cataract task label for the corresponding original eye image; the lens opacity data and the cataract category information are determined by observation results of the corresponding sample eyes using a slit-lamp microscope.

[0230] The dataset acquisition unit is used to obtain datasets from different domains using the original eye image labeled with the refractive power task label, the pupil mask image labeled with the segmentation task label and the refractive power task label, and the original eye image labeled with the cataract task label.

[0231] In some other embodiments, the multi-task image recognition model includes a shared encoder and decoders corresponding to multiple tasks belonging to different domains; based on this, the task reference data acquisition module 92 may include:

[0232] A shared feature data acquisition unit is used to input the image to be identified into the shared encoder for encoding processing to obtain shared feature data for the multiple tasks; the shared feature data can characterize the hidden relationships between the multiple tasks;

[0233] The task reference data acquisition unit is used to input the shared feature data into the decoders corresponding to the multiple tasks respectively for processing, so as to obtain the task reference data of the eye to be identified for the multiple tasks in different fields.

[0234] The network structures of the decoders corresponding to the multiple tasks are different, and the network structure of the decoders is determined according to the corresponding task type.

[0235] Optionally, the shared feature data acquisition unit may include:

[0236] The first feature map obtaining unit is used to input the image to be recognized into the shared encoder, extract features through a convolutional layer of the first scale, and obtain the first feature map after processing by a max pooling layer;

[0237] The second feature map obtaining unit is used to sequentially input the first feature map into the second-scale convolutional layer and the first-scale convolutional layer for feature extraction. After processing by the max pooling layer, the second feature map is obtained.

[0238] The shared feature data acquisition unit is used to sequentially input the second feature map into two convolutional layers of the first scale for feature extraction, and process the obtained third feature map through a max pooling layer to obtain shared feature data for the multiple tasks.

[0239] In this configuration, the convolution kernels in the first-scale convolutional layer perform convolution operations according to a first step length, and the convolution kernels in the second-scale convolutional layer perform convolution operations according to a second step length. The first step length is smaller than the second step length, and the first scale of the convolution kernel is smaller than the second scale.

[0240] Optionally, the above-mentioned device may further include:

[0241] An image size determination module is used to determine whether the image size of the eye image to be identified meets the input image size requirements of the shared encoder;

[0242] The image processing module is used to process the eye image to be identified according to the input image size requirements when the determination result of the image size determination module is not met.

[0243] Based on this, the first feature map obtaining unit mentioned above may include:

[0244] An image input unit is used to input the processed eye image to be recognized, which meets the input image size requirements, into the shared encoder.

[0245] In some other embodiments, the task reference data acquisition unit described above may include:

[0246] The first decoding unit is used to input the shared feature data into a regression decoder for the refractive power recognition task, and obtain refractive reference data of the eye to be identified through regression processing of two fully connected layers in the regression decoder; the refractive reference data includes spherical power and cylindrical power characterizing the refractive power of the eye to be identified;

[0247] The second decoding unit is used to input the shared feature data into a classification decoder for the cataract identification task. After classification processing by two fully connected layers in the classification decoder, the classification result is normalized to obtain the predicted probability that the eye to be identified belongs to the cataract category, and / or the lens opacity data of the eye to be identified; the lens opacity data can characterize the cataract level of the eye to be identified.

[0248] The third decoding unit is used to input the shared feature data into the segmentation decoder for the pupil segmentation task, extract features through convolutional layers of different scales in the segmentation decoder, process the feature map output by the connected convolutional layers through multiple upsampling layers or feature joint layers, input the processed feature map into the unit convolutional layer for feature data processing, and obtain the pupil region mask of the eye to be identified.

[0249] The convolutional layers of different scales include multiple convolutional layers of the first scale and one convolutional layer of the second scale; the feature fusion layer is connected to the corresponding convolutional layer of the shared encoder to realize the feature fusion operation between the feature map output by the convolutional layer and the feature map output by the convolutional layer connected in the segmentation decoder, so as to increase the low-order feature data lost by the feature extraction process of the convolutional layer connected in the segmentation decoder.

[0250] Combining the datasets from different domains discussed above, the model training module for multi-task image recognition models, obtained from the acquired datasets from different domains, can include:

[0251] A cross-domain feature extraction unit is used to input images from datasets of different domains into an initial shared encoder for feature extraction, resulting in first-domain feature data and second-domain feature data.

[0252] The MMD processing unit is used to minimize the distribution difference between the first domain feature data and the second domain feature data to obtain sample feature data for the corresponding sample eye, and the maximum mean difference MMD loss between the first domain feature data and the second domain feature data.

[0253] The sample reference data acquisition unit is used to input the sample feature data into the initial regression decoder for the refractive error recognition task, the initial classification decoder for the cataract recognition task, and the initial segmentation decoder for the pupil segmentation task, respectively, to obtain the sample reference data for the corresponding task.

[0254] The task loss acquisition unit is used to acquire the task loss between different sample reference data and the corresponding task labels annotated in the sample image;

[0255] The network parameter adjustment unit is used to adjust the network parameters of the initial shared encoder, the initial regression decoder, the initial classification decoder and the initial segmentation decoder respectively based on the weighted total loss of the MMD loss and the multiple task losses, so as to learn and train the image recognition model with the adjusted network parameters through the images in the dataset.

[0256] The multi-task image recognition model acquisition unit is used to determine that the weighted total loss satisfies the training constraints, and to obtain the multi-task image recognition model by using the finally trained shared encoder and the regression decoder, classification decoder and segmentation decoder for different tasks.

[0257] Based on the description of the above embodiments, the task reference data output module 93 may include at least one of the following output units:

[0258] The first output unit is used to send the eye image to be identified to a display for display, and to display the task reference data of each of the plurality of tasks on the eye image to be identified;

[0259] The second output unit is used to generate an eye detection report containing task reference data for each of the multiple tasks according to a preset output format, and output the eye detection report.

[0260] The third output unit is used to process the task reference data of each of the multiple tasks according to the task identification rules of the different domains, and output the corresponding task identification results.

[0261] It should be noted that the various modules and units in the above-mentioned device embodiments can all be stored in the memory as program modules. The processor executes the above-mentioned program modules stored in the memory to realize the corresponding functions. The functions realized by each program module and its combination, as well as the technical effects achieved, can be referred to the description of the corresponding part of the above-mentioned method embodiments. This embodiment will not repeat them here.

[0262] The present invention also provides a computer-readable storage medium on which a computer program can be stored, which can be called and loaded by a processor to implement the various steps of the eye image recognition method based on multi-task learning described in the above embodiments. The specific implementation process can be referred to the description of the corresponding part of the above embodiments, and will not be repeated in this embodiment.

[0263] Reference Figure 10 The above is a schematic diagram of the hardware structure of an optional example of a computer device suitable for the multi-task learning-based eye image recognition method proposed in this application. Figure 10 As shown, the computer device may include: a communication interface 101, a memory 102, and a processor 103, wherein:

[0264] The number of communication interface 101, memory 102 and processor 103 can all be at least one, and communication interface 101, memory 102 and processor 103 can all be connected to a communication bus to realize data interaction between them through the communication bus. The specific implementation process can be determined according to the needs of the specific application scenario, and will not be described in detail in this invention.

[0265] The communication interface 101 may include communication interfaces capable of data interaction using wireless communication networks, such as communication interfaces corresponding to WIFI modules, 5G / 6G (fifth-generation mobile communication network / sixth-generation mobile communication network) modules, GPRS modules, etc., to realize wireless communication between the computer device and other devices. Of course, the communication interface 101 may also be a data interface for wired communication, allowing the computer device to connect to other devices via a corresponding type of data cable to achieve data transmission between different devices. It should be understood that the aforementioned communication interface 101 includes communication interfaces for data interaction between internal components of the computer device, such as USB interfaces, serial / parallel ports, multimedia transmission interfaces, etc. This application does not limit the type or number of communication interfaces 101.

[0266] In this embodiment, the memory 102 can be used to store a program that implements the eye image recognition method based on multi-task learning described in any of the above method embodiments; the processor 103 can load and execute the program stored in the memory 102 to implement each step of the eye image recognition method based on multi-task learning proposed in any of the above method embodiments of this application. The specific implementation process can be referred to the description of the corresponding part of the corresponding embodiment above, and will not be repeated here.

[0267] In practical applications, memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device. Processor 103 may be a central processing unit (CPU), application-specific integrated circuit (ASIC), digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic device, etc., and the types of memory 102 and processor 103 can be determined according to actual needs.

[0268] It should be understood that, Figure 10 The structure of the computer device shown does not constitute a limitation on the computer device in the embodiments of the present invention. In practical applications, the computer device may include more than Figure 10 The additional components, or combinations thereof, shown may be determined based on the product type of the computer equipment. In the case of a terminal device, such as the aforementioned eccentric radiography equipment or other medical testing equipment, for example... Figure 11 As shown, the computer device may also include: an image acquisition component 104 for acquiring images of the eye to be identified; a display 105 for outputting task reference data for multiple tasks; a speaker 106 and at least one output component; and input components such as a microphone 107 and function buttons as needed to meet different input operations on the computer device; as well as a sensor module containing multiple sensors, a power supply component, etc., which can be determined according to the application scenario and are not listed here.

[0269] Finally, it should be noted that the terms "system," "device," "unit," and / or "module" used in this invention are methods for distinguishing different components, elements, parts, sections, or assemblies at different levels. However, if other terms can achieve the same purpose, they can be replaced by other expressions.

[0270] As indicated in this invention and the claims, unless the context clearly indicates otherwise, the words "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified steps and elements, which do not constitute an exclusive list, and the method or apparatus may also include other steps or elements. An element defined by the phrase "comprising an..." does not exclude the presence of other identical elements in the process, method, product, or apparatus that includes the element.

[0271] In the description of the embodiments of the present invention, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, in the description of the embodiments of the present invention, "multiple" refers to two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0272] Furthermore, the various embodiments in this specification are described in a progressive or parallel manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus and computer devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0273] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An eye image recognition method based on multi-task learning, characterized in that, The method includes: Obtain an image of the eye to be identified; Based on the trained multi-task image recognition model, the eye image to be identified is processed to obtain task reference data for multiple tasks in different domains of the eye to be identified; wherein, the multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different domains, the datasets in different domains include different eye images with corresponding task labels, and the multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task. Output task reference data for each of the multiple tasks to assist in determining the task recognition results of the eye to be identified in different domains; The multi-task image recognition model includes a shared encoder and decoders corresponding to multiple tasks belonging to different domains. The step of processing the eye image to be recognized based on the trained multi-task image recognition model to obtain task reference data for the eye to be recognized for multiple tasks in different domains includes: The eye image to be identified is input into the shared encoder, and features are extracted through a convolutional layer of the first scale. After processing by a max pooling layer, a first feature map is obtained. The first feature map is sequentially input into the second-scale convolutional layer and the first-scale convolutional layer for feature extraction. After processing by the max pooling layer, the second feature map is obtained. The second feature map is sequentially input into two convolutional layers of the first scale for feature extraction. After processing the obtained third feature map by a max pooling layer, shared feature data for the multiple tasks is obtained. Specifically, the convolutional kernels in the first-scale convolutional layers perform convolution operations with a first step size, and the convolutional kernels in the second-scale convolutional layers perform convolution operations with a second step size. The first step size is smaller than the second step size, and the first scale of the convolutional kernels is smaller than the second scale. The shared feature data can characterize the hidden relationships between the multiple tasks. The shared feature data is input into the decoders corresponding to the multiple tasks for processing to obtain the task reference data of the eye to be identified for each of the multiple tasks in different domains; the network structure of the decoders corresponding to the multiple tasks is different, and the network structure of the decoders is determined according to the corresponding task type.

2. The method according to claim 1, characterized in that, Obtaining the datasets from the different domains includes: Original eye images of multiple sample eyes were acquired; the original eye images were obtained by an eccentric photorefractive device. The original eye image is analyzed to obtain the refractive data corresponding to the sample eye, and the refractive data is determined as the refractive power task label of the original eye image; the refractive data includes spherical power and cylindrical power; The original eye image is segmented to obtain the pupil mask image and pupil positioning information corresponding to the sample eye. The pupil positioning information is then used as the segmentation task label for the original eye image. Acquire lens opacity data and / or cataract category information for each of the multiple sample eyes, and determine the lens opacity data and / or the cataract category information as the cataract task label for the corresponding original eye image; the lens opacity data and the cataract category information are determined by observation results of the corresponding sample eyes using a slit-lamp microscope; Data sets from different domains are obtained using the original eye image labeled with the refractive power task label, the pupil mask image labeled with the segmentation task label and the refractive power task label, and the original eye image labeled with the cataract task label.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Determine whether the image size of the eye image to be identified meets the input image size requirements of the shared encoder; If it does not meet the requirements, the eye image to be identified shall be processed according to the input image size requirements; The step of inputting the eye image to be identified into the shared encoder includes: The processed eye image that meets the input image size requirements is input into the shared encoder.

4. The method according to claim 1 or 2, characterized in that, The process of inputting the shared feature data into the decoders corresponding to the multiple tasks for processing to obtain task reference data for the eye to be identified in the multiple tasks in different domains includes: The shared feature data is input into a regression decoder for the refractive error recognition task. Through regression processing with two fully connected layers, refractive reference data of the eye to be identified is obtained. The refractive reference data includes spherical power and cylindrical power, which characterize the refractive error of the eye to be identified. The shared feature data is input into a classification decoder for the cataract identification task. After classification processing through two fully connected layers, the classification results are normalized to obtain the predicted probability that the eye to be identified belongs to the cataract category, and / or the lens opacity data of the eye to be identified; the lens opacity data can characterize the cataract level of the eye to be identified. The shared feature data is input into the segmentation decoder for the pupil segmentation task. After feature extraction through convolutional layers of different scales, the feature maps output by the connected convolutional layers are processed through multiple upsampling layers or feature joint layers. The processed feature maps are then input into a unit convolutional layer for feature data processing to obtain the pupil region mask of the eye to be identified. The convolutional layers of different scales include multiple convolutional layers of the first scale and one convolutional layer of the second scale; the feature fusion layer is connected to the corresponding convolutional layer of the shared encoder to realize the feature fusion operation between the feature map output by the convolutional layer and the feature map output by the convolutional layer connected in the segmentation decoder, so as to increase the low-order feature data lost by the feature fusion layer in the feature extraction process of the convolutional layer connected in the segmentation decoder.

5. The method according to claim 2, characterized in that, The process of performing multi-task learning based on datasets from different domains to obtain a multi-task image recognition model includes: Images from datasets in different domains are input into an initial shared encoder for feature extraction, resulting in first-domain feature data and second-domain feature data. Minimize the distribution difference processing of the first domain feature data and the second domain feature data to obtain sample feature data for the corresponding sample eyes, and the maximum mean difference MMD loss between the first domain feature data and the second domain feature data; The sample feature data are respectively input into the initial regression decoder for the refractive error recognition task, the initial classification decoder for the cataract recognition task, and the initial segmentation decoder for the pupil segmentation task for processing to obtain the sample reference data for the corresponding task. Obtain the task loss between different sample reference data and the corresponding task labels of the input image annotation; Based on the weighted total loss of the MMD loss and the losses of the multiple tasks, the network parameters of the initial shared encoder, the initial regression decoder, the initial classification decoder and the initial segmentation decoder are adjusted so as to learn and train the shared encoder with the adjusted network parameters and the decoders of the multiple tasks using images in the dataset. The weighted total loss is determined to satisfy the training constraints. Using the finally trained shared encoder and the regression decoder, classification decoder and segmentation decoder for different tasks, a multi-task image recognition model is obtained.

6. The method according to claim 1 or 2, characterized in that, The output of task reference data for each of the multiple tasks includes at least one of the following implementation methods: The eye image to be identified is sent to a display for display, and the task reference data of each of the multiple tasks is displayed on the eye image to be identified; According to the preset output format, generate an eye detection report containing task reference data for each of the multiple tasks, and output the eye detection report; According to the task identification rules of the different fields, the task reference data of each of the multiple tasks are processed, and the corresponding task identification results are output.

7. An eye image recognition device based on multi-task learning, characterized in that, The device includes An eye image acquisition module is used to acquire an image of the eye to be identified. The task reference data acquisition module is used to process the eye image to be identified based on the trained multi-task image recognition model to obtain task reference data for the eye to be identified for multiple tasks in different domains; wherein, the multi-task image recognition model is obtained by multi-task learning based on the obtained datasets in different domains, the datasets in different domains include different eye images with corresponding task labels, and the multiple tasks include refractive error recognition task, pupil segmentation task and cataract recognition task. The task reference data output module is used to output the task reference data of each of the multiple tasks to help determine the task recognition results of the eye to be recognized in different fields; The multi-task image recognition model includes a shared encoder and decoders corresponding to multiple tasks belonging to different domains. The task reference data acquisition module includes: The first feature map obtaining unit is used to input the eye image to be identified into the shared encoder, extract features through a convolutional layer of the first scale, and obtain the first feature map after processing by a max pooling layer. The second feature map obtaining unit is used to sequentially input the first feature map into the second-scale convolutional layer and the first-scale convolutional layer for feature extraction. After processing by the max pooling layer, the second feature map is obtained. A shared feature data acquisition unit is used to sequentially input the second feature map into two convolutional layers of the first scale for feature extraction, and then process the obtained third feature map through a max pooling layer to obtain shared feature data for the multiple tasks; wherein, the convolutional kernels in the first-scale convolutional layers perform convolution operations with a first step size, and the convolutional kernels in the second-scale convolutional layers perform convolution operations with a second step size, wherein the first step size is smaller than the second step size, and the first scale of the convolutional kernels is smaller than the second scale; the shared feature data can characterize the hidden relationships between the multiple tasks. The task reference data acquisition unit is used to input the shared feature data into the decoders corresponding to the multiple tasks respectively for processing, so as to obtain the task reference data of the eye to be identified for the multiple tasks in different fields; the network structure of the decoders corresponding to the multiple tasks is different, and the network structure of the decoder is determined according to the corresponding task type.

8. A computer device, characterized in that, The computer device includes: Communication interface; A memory for storing a program that implements the eye image recognition method based on multi-task learning as described in any one of claims 1-6; A processor is configured to load and execute a program stored in the memory to implement the eye image recognition method based on multi-task learning as described in any one of claims 1-6.

Images