System for detecting sperm in azoospermic samples

Abstract

The present disclosure describes a system for detecting sperm. The system comprises a computer system (1300) and a microscope (110) that are in communication with each other. The computer system comprises a processor (1305); and memory (1306) in communication with the processor. The memory comprises a computer application program (1333) that is executable by the processor to perform a method (400) for receiving (430) an image of a portion of a sample; detecting (200) sperm in the image using a model trained for detecting sperm; and indicating the sperm detected in the image. The microscope has a camera, the microscope being configured to capture an image of the portion of the sample and to transmit the captured image to the processor.

Description

SYSTEM FOR DETECTING SPERM IN AZOOSPERMIC SAMPLESRelated Application

[0001] The present application relates to Australian Patent Application No. 2024219482, filed on 6 September 2024, the content of which is incorporated herein by reference in its entirety.Technical Field

[0002] The present invention relates generally to assisted reproduction and, in particular, to detecting sperm in azoospermic samples for intracytoplasmic sperm injection (ICSI) purposes. The present invention also relates to a method and apparatus for detecting sperm in azoospermic samples, and to a computer program product including a computer readable medium having recorded thereon a computer program for controlling a microscope to capture images of portions of the azoospermic samples and to detect sperm in the captured images.Background

[0003] Azoospermia, defined as the absence of spermatozoa in centrifuged semen on at least two occasions, is the most severe form of male infertility, affecting 10-20% of infertile men and 1% of the general male population. Azoospermia can be classified as either obstructive and / or non-obstructive. Obstructive azoospermia (OA) occurs due to obstruction of the reproductive tract and constitutes 40% of azoospermic cases, while non-obstructive azoospermia (NOA) results from primary, secondary, or incomplete / ambiguous testicular failure which compromises sperm production and constitutes 60% of azoospermic cases. Patients OA may be remedied with reconstruction of the reproduction tract (e.g., vasovasostomy, vasoepididymostomy, or transurethral resection ejaculatory duct). Alternatively, the sperm of OA patients may be extracted from the testis via testicular sperm aspiration (TESA), testicular sperm extraction (TESE), microdissection testicular sperm extraction (mTESE), the epididymis Microsurgical epididymal sperm aspiration (MESA), or percutaneous epididymal sperm aspiration (PESA). The sperm of NOA patients may also be extracted from the testis using the above noted methods of TESA, TESE, or Micro TESE. The surgically collected sperm is then used for ICSI.

[0004] The gold-standard for treating NOA patients is mTESE with a high sperm retrieval rate of up to 64%. Although these rates seem promising, the current manual examination process to find sperm within tissue recovered from mTESE surgeries is time-consuming and inefficient, typically taking anywhere between 1 to 6 hours of laboratory time, and in some cases even upAH25(P0057851 PCT_46534242_1 DOCX)to 14 hours. This extended time is due to the requirement for manual searching through prepared suspensions of testicular tissue with a microscope, before using the isolated sperm for ICSI.

[0005] The outcome of such searching is dependent upon the complexity and contamination of the prepared suspensions of testicular tissue. Further, viable sperm are easily overlooked due to variables such as collateral cell density, resulting in a process that is prone to human error, combined with inexperience and fatigue of laboratory staff. For NOA patients, if sperm are overlooked due to human error, this could incorrectly indicate absolute infertility. Similarly, for extended sperm searches in semen as a diagnostic or as a last check in the ejaculate before surgery, failure to identify sperm could striate patients into surgery unnecessarily. Furthermore, prolonged sample examination procedures can have adverse effects on the viability of sperm, consequently affecting their potential for fertilization.

[0006] There is therefore a need for a more efficient and higher throughput method capable of locating and isolating sperm from the prepared suspension.Summary

[0007] It is an object of the present invention to substantially overcome, or at least ameliorate, one or more disadvantages of existing arrangements.

[0008] Disclosed are arrangements which seek to address the above problems by providing a system for capturing images of portions of azoospermic samples and processing the images to determine the existence of sperm in the captured images.

[0009] According to an aspect of the present disclosure, there is provided a method of detecting sperm, comprising: receiving an image of a portion of a sample; detecting sperm in the image using a model trained for detecting sperm; and indicating the sperm detected in the image.

[0010] Another aspect of the present disclosure provides a system for detecting sperm, the system comprising: a computer system comprising: a processor; and memory in communication with the processor, the memory comprising a computer application program that is executable by the processor to perform a method comprising; receiving an image of a portion of a sample; detecting sperm in the image using a model trained for detecting sperm; and indicating the sperm detected in the image; and a microscope having a camera, wherein the microscope is inAH25(P0057851 PCT_46534242_1 DOCX)communication with the processor, the microscope being configured to capture an image of the portion of the sample and to transmit the captured image to the processor.

[0011] According to another aspect of the present disclosure, there is provided an apparatus for implementing any one of the aforementioned methods.

[0012] According to another aspect of the present disclosure, there is provided a computer program product including a computer readable medium having recorded thereon a computer program for implementing any one of the methods described above.

[0013] Other aspects are also disclosed.Brief Description of the Drawings

[0014] At least one embodiment of the present invention will now be described with reference to the drawings, in which:

[0015] Figs. 1A and 1 B form a schematic block diagram of a system upon which arrangements described can be practiced;

[0016] Fig. 2 is a flow diagram of a method for detecting sperm in an image;

[0017] Fig. 3 is a flow diagram of a method for detecting sperm in a sample using the system of Figs. 1A and 1B;

[0018] Fig. 4 is a flow diagram of a sub-process of the method shown in Fig. 3;

[0019] Fig. 5 is a flow diagram of a method of preparing the sample for use in the method ofFig. 3;

[0020] Figs. 6 and 7 show the results of sperm detection using conventional approaches and the method of Figs. 2 to 5; and

[0021] Fig. 8 shows a flow diagram of a method for training a model used for performing the method of Fig. 2.Detailed DescriptionAH25(P0057851 PCT_46534242_1 DOCX)

[0022] Where reference is made in any one or more of the accompanying drawings to steps and / or features, which have the same reference numerals, those steps and / or features have for the purposes of this description the same function(s) or operation(s), unless the contrary intention appears.

[0023] It is to be noted that the discussions contained in the "Background" section and that above relating to prior art arrangements relate to discussions of documents or devices which form public knowledge through their respective publication and / or use. Such should not be interpreted as a representation by the present inventor(s) or the patent applicant that such documents or devices in any way form part of the common general knowledge in the art.Computer Description

[0024] Figs. 1A and 1 B depict a general-purpose computer system 1300, upon which the various arrangements described can be practiced.

[0025] As seen in Fig. 1A, the computer system 1300 includes: a computer module 1301; input devices such as a keyboard 1302, a mouse pointer device 1303, a scanner 1326, a camera 1327, and a microphone 1380; and output devices including a printer 1315, a display device 1314 and loudspeakers 1317. An external Modulator-Demodulator (Modem) transceiver device 1316 may be used by the computer module 1301 for communicating to and from a communications network 1320 via a connection 1321. The communications network 1320 may be a wide-area network (WAN), such as the Internet, a cellular telecommunications network, or a private WAN. Where the connection 1321 is a telephone line, the modem 1316 may be a traditional “dial-up” modem. Alternatively, where the connection 1321 is a high capacity (e.g., cable) connection, the modem 1316 may be a broadband modem. A wireless modem may also be used for wireless connection to the communications network 1320.

[0026] The computer module 1301 typically includes at least one processor unit 1305, and a memory unit 1306. For example, the memory unit 1306 may have semiconductor random access memory (RAM) and semiconductor read only memory (ROM). The computer module 1301 also includes an number of input / output (I / O) interfaces including: an audio-video interface 1307 that couples to the video display 1314, loudspeakers 1317 and microphone 1380; an I / O interface 1313 that couples to the keyboard 1302, mouse 1303, scanner 1326, camera 1327 and optionally a joystick or other human interface device (not illustrated); and an interface 1308 for the external modem 1316 and printer 1315. In some implementations, the modem 1316 may be incorporated within the computer module 1301, forAH25(P0057851 PCT_46534242_1 DOCX)example within the interface 1308. The computer module 1301 also has a local network interface 1311, which permits coupling of the computer system 1300 via a connection 1323 to a local-area communications network 1322, known as a Local Area Network (LAN). As illustrated in Fig. 1A, the local communications network 1322 may also couple to the wide network 1320 via a connection 1324, which would typically include a so-called “firewall” device or device of similar functionality. The local network interface 1311 may comprise an Ethernet circuit card, a Bluetooth® wireless arrangement or an IEEE 802.11 wireless arrangement; however, numerous other types of interfaces may be practiced for the interface 1311.

[0027] Fig. 1A shows that a microscope 110 is connected to the wide network 1320, such that the microscope 110 is connected to the processor 1305. The microscope 110 includes a camera (not shown) and a receptacle for receiving a dish (not shown). When a dish with azoospermic sample is disposed on the receptacle, the camera can capture images of the sample.

[0028] The processor 1305 transmits control signals to the microscope 110 and receives images from the microscope 110. Although the microscope 110 is shown to be connected to the wide network 1320, the microscope 110 may be connected to the local area network 1322 or the I / O interface 1308 to receive instructions from the processor 1305 and to transmit images to the processor 1305. The control signals from the processor 1305 to the microscope 110 are to adjust the camera position or camera focal length of the microscope 110, the settings (e.g., brightness) of the microscope 110, magnification level of the camera, camera resolution, the frame rate cap, colour saturation, hue adjustment, sharpening, thresholding, resizing, and the like.

[0029] The I / O interfaces 1308 and 1313 may afford either or both of serial and parallel connectivity, the former typically being implemented according to the Universal Serial Bus (USB) standards and having corresponding USB connectors (not illustrated). Storage devices 1309 are provided and typically include a hard disk drive (HDD) 1310. Other storage devices such as a floppy disk drive and a magnetic tape drive (not illustrated) may also be used. An optical disk drive 1312 is typically provided to act as a non-volatile source of data. Portable memory devices, such optical disks (e.g., CD-ROM, DVD, Blu-ray Disc™), USB-RAM, portable, external hard drives, and floppy disks, for example, may be used as appropriate sources of data to the system 1300.

[0030] The components 1305 to 1313 of the computer module 1301 typically communicate via an interconnected bus 1304 and in a manner that results in a conventional mode of operation ofAH25(P0057851 PCT_46534242_1 DOCX)the computer system 1300 known to those in the relevant art. For example, the processor 1305 is coupled to the system bus 1304 using a connection 1318. Likewise, the memory 1306 and optical disk drive 1312 are coupled to the system bus 1304 by connections 1319. Examples of computers on which the described arrangements can be practised include IBM-PC’s and compatibles, Sun Sparcstations, Apple Mac™ or like computer systems.

[0031] The method of detecting sperm in an image or in a sample may be implemented using the computer system 1300 wherein the processes of Figs. 2 to 4, to be described, may be implemented as one or more software application programs 1333 executable within the computer system 1300. In particular, the steps of the method of detecting sperm in an image or in a sample are effected by instructions 1331 (see Fig. 1 B) in the software 1333 that are carried out within the computer system 1300. The software instructions 1331 may be formed as one or more code modules, each for performing one or more particular tasks. The software may also be divided into two separate parts, in which a first part and the corresponding code modules performs the sperm detection methods and a second part and the corresponding code modules manage a user interface between the first part and the user. Similarly, the method of training a model for detecting sperm shown in Fig. 8, to be described, may be implemented as one or more software application programs 1333 executable within the computer system 1300.

[0032] The software may be stored in a computer readable medium, including the storage devices described below, for example. The software is loaded into the computer system 1300 from the computer readable medium, and then executed by the computer system 1300. A computer readable medium having such software or computer program recorded on the computer readable medium is a computer program product. The use of the computer program product in the computer system 1300 preferably effects an advantageous apparatus for detecting sperm in an image or a sample.

[0033] The software 1333 is typically stored in the HDD 1310 or the memory 1306. The software is loaded into the computer system 1300 from a computer readable medium, and executed by the computer system 1300. Thus, for example, the software 1333 may be stored on an optically readable disk storage medium (e.g., CD-ROM) 1325 that is read by the optical disk drive 1312. A computer readable medium having such software or computer program recorded on it is a computer program product. The use of the computer program product in the computer system 1300 preferably effects an apparatus for detecting sperm in an image or a sample.AH25(P0057851 PCT_46534242_1 DOCX)

[0034] In some instances, the application programs 1333 may be supplied to the user encoded on one or more CD-ROMs 1325 and read via the corresponding drive 1312, or alternatively may be read by the user from the networks 1320 or 1322. Still further, the software can also be loaded into the computer system 1300 from other computer readable media.Computer readable storage media refers to any non-transitory tangible storage medium that provides recorded instructions and / or data to the computer system 1300 for execution and / or processing. Examples of such storage media include floppy disks, magnetic tape, CD-ROM, DVD, Blu-ray™ Disc, a hard disk drive, a ROM or integrated circuit, USB memory, a magnetooptical disk, or a computer readable card such as a PCMCIA card and the like, whether or not such devices are internal or external of the computer module 1301. Examples of transitory or non-tangible computer readable transmission media that may also participate in the provision of software, application programs, instructions and / or data to the computer module 1301 include radio or infra-red transmission channels as well as a network connection to another computer or networked device, and the Internet or Intranets including e-mail transmissions and information recorded on Websites and the like.

[0035] The second part of the application programs 1333 and the corresponding code modules mentioned above may be executed to implement one or more graphical user interfaces (GUIs) to be rendered or otherwise represented upon the display 1314. Through manipulation of typically the keyboard 1302 and the mouse 1303, a user of the computer system 1300 and the application may manipulate the interface in a functionally adaptable manner to provide controlling commands and / or input to the applications associated with the GUI(s). Other forms of functionally adaptable user interfaces may also be implemented, such as an audio interface utilizing speech prompts output via the loudspeakers 1317 and user voice commands input via the microphone 1380.

[0036] Fig. 1 B is a detailed schematic block diagram of the processor 1305 and a “memory” 1334. The memory 1334 represents a logical aggregation of all the memory modules (including the HDD 1309 and semiconductor memory 1306) that can be accessed by the computer module 1301 in Fig. 1A.

[0037] When the computer module 1301 is initially powered up, a power-on self-test (POST) program 1350 executes. The POST program 1350 is typically stored in a ROM 1349 of the semiconductor memory 1306 of Fig. 1A. A hardware device such as the ROM 1349 storing software is sometimes referred to as firmware. The POST program 1350 examines hardware within the computer module 1301 to ensure proper functioning and typically checks the processor 1305, the memory 1334 (1309, 1306), and a basic input-output systems softwareAH25(P0057851 PCT_46534242_1 DOCX)(BIOS) module 1351, also typically stored in the ROM 1349, for correct operation. Once the POST program 1350 has run successfully, the BIOS 1351 activates the hard disk drive 1310 of Fig. 1A. Activation of the hard disk drive 1310 causes a bootstrap loader program 1352 that is resident on the hard disk drive 1310 to execute via the processor 1305. This loads an operating system 1353 into the RAM memory 1306, upon which the operating system 1353 commences operation. The operating system 1353 is a system level application, executable by the processor 1305, to fulfil various high level functions, including processor management, memory management, device management, storage management, software application interface, and generic user interface.

[0038] The operating system 1353 manages the memory 1334 (1309, 1306) to ensure that each process or application running on the computer module 1301 has sufficient memory in which to execute without colliding with memory allocated to another process. Furthermore, the different types of memory available in the system 1300 of Fig. 1A must be used properly so that each process can run effectively. Accordingly, the aggregated memory 1334 is not intended to illustrate how particular segments of memory are allocated (unless otherwise stated), but rather to provide a general view of the memory accessible by the computer system 1300 and how such is used.

[0039] As shown in Fig. 1 B, the processor 1305 includes a number of functional modules including a control unit 1339, an arithmetic logic unit (ALU) 1340, and a local or internal memory 1348, sometimes called a cache memory. The cache memory 1348 typically includes a number of storage registers 1344 - 1346 in a register section. One or more internal busses 1341 functionally interconnect these functional modules. The processor 1305 typically also has one or more interfaces 1342 for communicating with external devices via the system bus 1304, using a connection 1318. The memory 1334 is coupled to the bus 1304 using a connection 1319.

[0040] The application program 1333 includes a sequence of instructions 1331 that may include conditional branch and loop instructions. The program 1333 may also include data 1332 which is used in execution of the program 1333. The instructions 1331 and the data 1332 are stored in memory locations 1328, 1329, 1330 and 1335, 1336, 1337, respectively. Depending upon the relative size of the instructions 1331 and the memory locations 1328-1330, a particular instruction may be stored in a single memory location as depicted by the instruction shown in the memory location 1330. Alternately, an instruction may be segmented into a number of parts each of which is stored in a separate memory location, as depicted by the instruction segments shown in the memory locations 1328 and 1329.AH25(P0057851 PCT_46534242_1 DOCX)

[0041] In general, the processor 1305 is given a set of instructions which are executed therein. The processor 1305 waits for a subsequent input, to which the processor 1305 reacts to by executing another set of instructions. Each input may be provided from one or more of a number of sources, including data generated by one or more of the input devices 1302, 1303, data received from an external source across one of the networks 1320, 1302, data retrieved from one of the storage devices 1306, 1309 or data retrieved from a storage medium 1325 inserted into the corresponding reader 1312, all depicted in Fig. 1A. The execution of a set of the instructions may in some cases result in output of data. Execution may also involve storing data or variables to the memory 1334.

[0042] The disclosed sperm detection arrangements use input variables 1354, which are stored in the memory 1334 in corresponding memory locations 1355, 1356, 1357. The sperm detection arrangements produce output variables 1361, which are stored in the memory 1334 in corresponding memory locations 1362, 1363, 1364. Intermediate variables 1358 may be stored in memory locations 1359, 1360, 1366 and 1367.

[0043] Referring to the processor 1305 of Fig. 1B, the registers 1344, 1345, 1346, the arithmetic logic unit (ALU) 1340, and the control unit 1339 work together to perform sequences of micro-operations needed to perform “fetch, decode, and execute” cycles for every instruction in the instruction set making up the program 1333. Each fetch, decode, and execute cycle comprises:

[0044] a fetch operation, which fetches or reads an instruction 1331 from a memory location 1328, 1329, 1330;

[0045] a decode operation in which the control unit 1339 determines which instruction has been fetched; and

[0046] an execute operation in which the control unit 1339 and / or the ALU 1340 execute the instruction.

[0047] Thereafter, a further fetch, decode, and execute cycle for the next instruction may be executed. Similarly, a store cycle may be performed by which the control unit 1339 stores or writes a value to a memory location 1332.

[0048] Each step or sub-process in the processes of Figs. 2 to 4 and 8 is associated with one or more segments of the program 1333 and is performed by the register section 1344,AH25(P0057851 PCT_46534242_1 DOCX)1345, 1347, the ALU 1340, and the control unit 1339 in the processor 1305 working together to perform the fetch, decode, and execute cycles for every instruction in the instruction set for the noted segments of the program 1333.

[0049] The method of sperm detection may alternatively be implemented in dedicated hardware such as one or more integrated circuits performing the functions or sub functions of the methods of Figs. 2 to 4 and 8. Such dedicated hardware may include graphic processors, digital signal processors, or one or more microprocessors and associated memories.

[0050] Fig. 2 shows a method 200 for detecting sperm in an image. The image is captured by the microscope 110 and is then provided to the processor 305. The method 200 is implemented as one or more software application programs 1333 executable within the computer system 1300. The method 200 commences when an image is received by the processor 305 of the computer system 1300. When the image is received, the method 200 starts with step 210 by dividing the image into cells. The method 200 then proceeds from step 210 to step 220.

[0051] In step 220, the method 200 extracts features (such as edges, shapes, textures, etc.) of shapes in each cell. The method 200 then proceeds from step 220 to step 230.

[0052] In step 230, the method 200 predicts a sperm in each cell based on the extracted features. The method 200 may be implemented using non-neural approaches (e.g., Viola- Jones object detection framework, scale-invariant feature transform, histogram of oriented gradients, etc.) or neural network approaches (e.g., region proposals, single shot multibox detector, retina net, You Only Look Once (YOLO) networks, faster R-CNN, Mask R-CNN, EfficientDet, Vision Transformer, etc.). In one arrangement, the method 200 includes multiple models of neural network approaches, where each model relates to a set of settings of the camera of the microscope 110. For example, a first neural model is trained to detect sperm with magnification 4x at a first focal length, while a second neural model is trained to detect sperm with magnification 8x at a second focal length. The different neural models enable the method 200 to use the best neural model to use at particular settings of the camera of the microscope 110.

[0053] These approaches can be used in step 230, which provides an output indicating whether a sperm has been detected within a cell and the chances that the prediction are correct. The output also provides a bounding box outlining the location of the predicted sperm. As described hereinafter in relation to Fig. 8, neural approaches may be trained to detect spermAH25(P0057851 PCT_46534242_1 DOCX)at a particular magnification level. If such is performed to train the neural model, then a particular neural model may be selected for a particular magnification level of the camera of the microscope 110. For example, a trained model for 10x magnification level is used when the camera is set to 10x magnification level.

[0054] The method 200 then proceeds from step 230 to step 240.

[0055] In step 240, the method 200 predicts a location of sperm in the image based on the predicted sperm in the cells of the image. The method 200 combines the cells together and determines whether the chances of sperm being detected in areas of images covered by multiple cells to be accurate or inaccurate. In one arrangement, the method 200 increases or decreases the chances of sperm being detected based on whether there are connected bounding boxes in adjacent cells. In another arrangement, the method 200 increases or decreases the chances of sperm being detected based on the predicted location of sperm over multiple images of the same location.

[0056] For example, if bounding boxes of two adjacent cells with chances of correct prediction to be above 50% are connected, then the method 200 combines the bounding boxes into one bounding box predicting that sperm is detected with updated chances of correct prediction to 75%. This step is also referred to as “Non-Maximum Suppression.”

[0057] The method 200 concludes at the conclusion of step 240.

[0058] At the conclusion of the method 200, locations of sperm in the image are marked by bounding boxes indicating detected sperm.

[0059] Fig. 3 shows a method 300 for detecting sperm in a sample. Certain parts of the method 300 are implemented as one or more software application programs 1333 executable within the computer system 1300. The method 300 commences at step 310 by calibrating the microscope 110. Step 310 may be performed a computer program connecting to the microscope 110 is turned on. Alternatively, step 310 may be performed when the microscope 110 is turned on and is connected to the computer system 1300. When turned on, the computer system 1300 transmits a control signal to the microscope 110 to capture an image. In turn, the microscope 110 captures the image and transmits the image to the computer system 1300. The computer system 1300 then determines the brightness and focus of the camera. In turn, the computer system 1300 transmits control signals to the microscope 110 to adjust the brightness and focus of the camera.AH25(P0057851 PCT_46534242_1 DOCX)

[0060] In one arrangement, the computer system 1300 transmits control signals to the microscope 110 to adjust the exposure of the camera to assist with controlling the brightness.

[0061] The method 300 then proceeds from step 310 to step 320.

[0062] In step 320, the microscope 110 receives a sample (which may be disposed on a dish or a glass slide under cover slip). A method 500 (shown in Fig. 5 and described below) shows a flow diagram for medical personnel to extract and process testicular tissue and to provide the prepared testicular tissue to the microscope 110. In other arrangements, the sample may be extracted from semen. Once the sample is received at the microscope 110, the method 300 then proceeds from step 320 to step 330.

[0063] In step 330, the microscope 110 is further calibrated. In one arrangement, the computer system 1300 determines whether the brightness and focus setting of the microscope 110 is suitable once the dish is disposed in the receptacle of the microscope 110. The computer system 1300 then transmits control signals to the microscope 110 to further adjust the brightness setting of the microscope 110 (if required). The method 300 then proceeds from step 330 to sub-process 400.

[0064] Sub-process 400 is shown in Fig. 4. Sub-process 400 is implemented as one or more software application programs 1333 executable within the computer system 1300. Sub-process 400 commences at step 410 by setting a position of the camera of the microscope 110. In one arrangement, the camera has predetermined positions (e.g., in mm increments) for a particular dish / glass slide size. These predetermined positions are pre-programmed so that the computer system 1300 can move the camera to any one of these predetermined positions. In another arrangement, the camera has a coordinate system that enables the camera to be moved within the coordinate system.

[0065] As described hereinafter, the camera is moved from one position to the next position so that images of the sample over the entirety of the dish are captured. Accordingly, the camera is initially set to a first predetermined position or to a first coordinate location. After the first or subsequent run, the camera is moved to the next predetermined position or the next coordinate location.

[0066] A skilled person in the art would understand that the location of the camera can be tracked by using appropriate parameters. Accordingly, methods for tracking the location of the camera and for moving the camera to a next position are not discussed in detail.AH25(P0057851 PCT_46534242_1 DOCX)

[0067] In step 410, the computer system 1300 transmits control signals to the microscope 110 to move the camera to a particular position.

[0068] Sub-process 400 then proceeds from step 410 to step 420.

[0069] In step 420, the focal length of the camera of the microscope 110 is set to a predetermined length. The focal length of the camera has predetermined lengths (e.g., in pm increments) for adjusting the focus of the camera.

[0070] The focal length of the camera is adjusted from one focal length to the next focal length so that images of a portion of the sample are captured with different focus. The different focus adjusts the quality of image being captured. Accordingly, the camera is initially set to a first focal length. After the first or subsequent run, the focal length of the camera is adjusted to the next focal length.

[0071] A skilled person in the art would understand that the current focal length of the camera can be tracked by using appropriate parameters. Accordingly, methods for tracking the current focal length of the camera and for moving the focal length of the camera to a next focal length are not discussed in detail.

[0072] In step 420, the computer system 1300 transmits control signals to the microscope 110 to set the focal length of the camera.

[0073] The sub-process 400 then proceeds from step 420 to step 430.

[0074] In step 430, the sub-process 400 receives an image of a portion of the sample at a particular location (as set by step 410) with a particular focal length (as set by step 420). The computer system in step 430 transmits control signals to the microscope 110 to capture an image. The microscope 110 in turn captures an image according to the control signals and transmits the captured image to the computer system 1300. The computer system 1300 therefore receives the captured image at step 430.

[0075] In one arrangement, the microscope 110 captures multiple images (e.g., a video) over a period of time and transmits the captured images to the computer system 1300.

[0076] The sub-process 400 has two loops, the first loop is to move the camera to different locations and the second loop is to adjust the focal length of the camera. The second loop isAH25(P0057851 PCT_46534242_1 DOCX)performed so that images are captured at different focal lengths of the camera at one location. Once the second loop is completed, the camera is moved to the next position so that images are captured at different focal lengths of the camera at the next location. This is repeated until images are captured for all locations of the camera. There are however scan options enabling a user of the computer system 1300 to elect a full scan (where both loops are performed), a quick scan (where only the first loop is performed), or an intermittent scan (where the sub-process 400 is paused when sperm is detected). The available scans are (1) full scan; (2) quick scan; and (3) intermittent scan. In full scan, the sub-process 400 is performed in its entirety such that both loops (i.e. , over the range of camera locations and camera focal lengths) are performed over the full dish. In quick scan, the sub-process 400 is performed by omitting the second loop (i.e., steps 420, 440, and 450). In intermittent scan, the sub-process 400 is performed by omitting the second loop and is paused every time sperm is detected.

[0077] When an image is received at step 430, the location and focal length of the camera are recorded and associated with the received image. Further, the received image may be displayed on the video display 1314. In the arrangement where multiple images are received by the computer system 1300, the computer system 1300 records and associates the multiple images with the location and focal length of the camera.

[0078] In one arrangement, a user may use the user inputs (e.g., 1302, 1303) to change the camera setting to capture images in grey scale. Setting the camera to grey scale desaturates the captured images to prevent colour variations affecting the method 200 in detecting sperm.

[0079] The sub-process 400 then proceeds from step 430 to the method 200. As described hereinbefore, the method 200 detects sperm within an image. In one arrangement, the method 200 selects one of the available neural models that is suitable for the current settings of the camera of the microscope 110. In this arrangement, when the first loop of the sub-process 400 is performed at a first focal length and a magnification level, the neural model trained for those settings is selected. Similarly, when the focal length is changed, another neural model trained for the amended settings is selected. Accordingly, the method 200 uses the best available neural model to detect the sperm.

[0080] Accordingly, the method 200 detects sperm of the image captured in step 430. The detected sperm is highlighted on the video display 1314 with a bounding box together with a value indicating the chances that the sperm have been correctly detected. The detected sperm is also associated with the processed image and the location and focal length of the camera. In one arrangement, the bounding box and the chances of accurate sperm detection are onlyAH25(P0057851 PCT_46534242_1 DOCX)updated periodically or at certain events (e.g., change of camera location). The periodical update is performed to reduce the frequency of the video display 1314 updating as the subprocess 400 is being executed (which reduces the flickering of the video display 1314 due to the high update frequency). The sub-process 400 then proceeds from the method 200 to step 440.

[0081] In step 440, the sub-process 400 determined whether all the focal lengths of the camera have been processed. If NOT, the sub-process 400 proceeds from step 440 to step 420 (where the focal length of the camera is adjusted to the next focal length). If YES, the subprocess 400 proceeds from step 440 to step 450.

[0082] In step 450, the sub-process 400 updates the detected sperm at that particular location of the camera based on the sperm detected on the images with different focal lengths. In one arrangement, the chances of sperm detected at that particular location are updated to the highest chance of sperm detected at the different focal lengths. In another arrangement, the chances of sperm detected at that particular location are updated to the average of the chances of sperm detected at the different focal lengths. The sub-process 400 then proceeds from step 450 to step 460.

[0083] In step 460, the sub-process 400 determined whether all the positions of the camera have been processed. If NOT, the sub-process 400 proceeds from step 460 to step 410 (where the position of the camera is moved to the next position). If YES, the sub-process 400 concludes at the conclusion of step 460.

[0084] At the conclusion of the sub-process 400, the computer system 1300 displays (at the video display 1314) an image, which is a combination of images stitched together from all the locations of the camera, displaying the sperm detected. The detected sperm are presented in bounding boxes together with the updated chances of accuracy (as determined at step 450 or by the method 200). In one arrangement, the images with the best quality are used to form the displayed image. In another arrangement, the images at one particular focal length are used to form the displayed image. In yet another arrangement, a list of detected sperm with the camera locations may also be displayed on the video display 1314, and the list is sorted according to the chances of accurate prediction. In yet another arrangement, notifications are generated to inform the user of sperm detected. These notifications may be a sound, a pointer, a flashing light, and the like in order to attract the user’s attention to sperm being detected.

[0085] As discussed in step 430, multiple images are captured over a duration of time (e.g., 5 seconds) in one arrangement. These images can then be used to determine whether detectedAH25(P0057851 PCT_46534242_1 DOCX)sperm are moving by determining whether the location and / or curve of detected sperm have changed in these images. For example, the sub-process 400 identifies a particular location to have detected sperm. The computer system 1300 then executes a computer program application running a method to determine whether the detected sperm is motile. To do so, the computer system 1300 processes the images captured at step 430 at that particular location at each focal length to determine whether the location and / or curve of the head or tail of the detected sperm have changed. If any changes are detected, then the computer system 1300 displays that the detected sperm may be motile.

[0086] In one arrangement, instead of executing sub-process 400, the user of the computer system use the user inputs 1302, 1303 to control the movement of the camera of the microscope 110, while the method 200 is being executed. In this arrangement, the method 200 displays any detected sperm on the video display 1314. However, if the user moves the camera too quickly, then the computer system 1300 displays an error informing the user that the movement of the camera is too quick for the sperm detection method 200 and the method 200 has therefore been turned off until the camera movement is within suitable speed.

[0087] In another arrangement, instead of executing sub-process 400, the user of the computer system use the user inputs 1302, 1303 to control the movement of the camera of the microscope 110, while the method 200 is being executed. In this arrangement, when the user moves the camera to a location, a video of the location is recorded and the recorded video is provided to the method 200. In turn, the method 200 provides a sound notification when sperm is detected to cue the user that sperm is detected on that particular location.

[0088] Fig. 5 shows a flow diagram of a method 500 performed by a user to prepare sample on a dish (which is to be disposed in the receptacle of the microscope 110, see step 320 of the method 300). In another arrangement, the sample is prepared on a glass slide, a glass slide under a cover slip, and the like. The method 500 commences at step 510 by retrieving testicular tissue from a patient using testicular biopsy (such as TESE, mTESE, and TESA methods). Surgical sperm extractions are performed under general anaesthesia. In one arrangement, the retrieved testicular tissue is immediately placed in a sterile conical tube containing a gamete buffer (e.g., 1 ml of GMOPS Plus (at 37°C)) and then transported to an assisted reproduction laboratory for processing. In another arrangement, the retrieved testicular tissue is processed in the surgical area. The gamete buffer is selected to support sperm vitality. The method 500 then proceeds from step 510 to step 520.AH25(P0057851 PCT_46534242_1 DOCX)

[0089] In step 520, the extracted testicular tissue is processed by a user. The extracted testicular tissue is placed in a gamete buffer (for example, 1-2 ml of G-MOPS Plus) in a sterile Petri dish under a stereo-microscope to wash off excess blood from the tissue. Similarly, the gamete buffer is selected to support sperm vitality.

[0090] In cases whereby imaging and / or testing is not possible on the same day or the following day, the samples are fixed with 4% formalin to preserve their morphological integrity and prevent any microbial growth.

[0091] The method 500 then proceeds from step 520 to step 530.

[0092] In step 530, the sample is prepared on a Petri dish with a gamete buffer (e.g., 300 ml of G-MOPS Plus). Long, flat droplets of the tissue suspension are placed on the dish and is then covered in media oil. Thinner droplets assist cells settle onto the bottom of the dish and into a single plane, making sperm detection easier. The tissue is then gently teased apart using sterile syringes to release potential spermatozoa from the tubules into the surrounding gamete buffer (e.g., G-MOPS Plus) medium. The macerated tissue and large pieces are then removed and placed into a separate tube, and the remaining suspension is used for the sperm search and treatment. The method 500 proceeds from step 530 to step 540.

[0093] In step 540, the dish is disposed in the receptacle of the microscope 110. Step 320 (from the point of view of the microscope 110) is equivalent to step 540 (from the point of view of the user). The method 500 then concludes at the conclusion of step 540.

[0094] The method 500 has been described in relation to extracting sample from testicular tissue. However, samples may also be extracted from semen (in which case steps 510 and 520 are not applicable), vaginal / rectal swab, and other locations where sperm may be located.

[0095] As described hereinbefore in relation to the method 200, there are non-neural and neural approaches for detecting sperm. In one arrangement, a neural approach using YOLO is used and the training of a YOLO model is described in relation to Fig. 8.

[0096] Fig. 8 shows a method 800 of training a YOLO model for detecting sperm in an image. The method 800 is implemented as one or more software application programs executable within a computer system. The YOLO model trained by the method 800 then performs the method 200 and is implemented as one or more software application programs 1333 executable within the computer system 1300.AH25(P0057851 PCT_46534242_1 DOCX)

[0097] The method 800 commences at step 810 by receiving images of samples. The samples are prepared in accordance with the method 500 and contain mixtures of spermatozoa, red blood cell, white blood cell, and epithelial cells from cell culture media. In one arrangement, 10 long, flat droplets of tissue suspension in G-MOPS Plus of 2-3 mm in length under media oil (e.g., OVOIL) are disposed on an ICSI dish (e.g., Vitrolife). CellSens Imaging software is used to capture images of the samples.

[0098] The training dataset comprised 540 images (152 from specifically prepared samples (i.e. , the samples prepared at step 530) and 388 from testicular tissue samples (i.e. , the samples acquired at step 520)). The 540 images contain 5624 unique sperm instances.

[0099] In one arrangement, further training dataset is provided from real samples to further improve the model for detecting sperm.

[0100] The method 800 then proceeds from step 810 to step 820.

[0101] In step 820, a user (via user inputs of the computer system executing the method 800) annotates the received images to indicate the sperm in the images. Images are annotated using the Computer Vision Annotation Tool (C AT). The annotation includes adding a bounding box around a sperm that enclose the entire visible spermatozoa including the head and tail. If the spermatozoon is partially occluded, the spermatozoa is still bound by a single bounding box encompassing all the visible areas.

[0102] In one arrangement, the further training dataset comprising real samples is annotated by a model that has been trained by the method 800.

[0103] The method 800 then proceeds from step 820 to step 830.

[0104] In step 830, the method 800 processes the annotated images. The images captured by CellSens Imaging software are 2456 x 1842px JPG images with 95% compression. JPG format is used as microscope 110 typically uses JPG format. The annotated images are resized to 1664 x 1664px with a black fill. The sperm instances in the images are then duplicated and augmented (e.g., flip, blur, darken, contrast, etc.) to generate at least one augmented copy per image, resulting in over 10,000 spermatozoa to train the YOLO model. Duplication is used to create more unique images for the YOLO model to learn from and is performed to improve dataset fidelity. By creating these augmented duplicate images, these images can be used in the training process as they may be considered functionally unique to their original copy.AH25(P0057851 PCT_46534242_1 DOCX)

[0105] Augmentations are applied to all images including duplicates from both the specifically prepared samples and the excess testicular tissue samples to inflate the training dataset and make the trained YOLO model more robust to variations in microscope camera images, such as compression artefacts, changing focal length or lighting and colour variations. In one arrangement, vertical flip is applied to each duplicate image, ensuring the duplicate image is uniquely different from its source, and then with various probabilities a series of augmentation techniques are employed using the Python-based Albumentations library. Initially, a blurring effect with a kernel size of 2 x 2 pixels is applied to each image to simulate the effect of slight defocusing. Thereafter, JPG compression is implemented, adjusting the compression quality to a range between 60% and 80%, to mimic the common lossy compression artefacts (features identifiable with the human eye) found in digital imaging. Other augmentations such as colouring (similar to the greying applied in step 430 of the method 400) and magnification (i.e., zoom in, zoom out) of the image may be applied.

[0106] The method 800 then proceeds from step 830 to step 840.

[0107] In step 840, the model is trained using the processed, annotated images. 85% of the images are used for training and 15% of the images are reserved for validation of the model’s performance after training. The YOLO model used here has a ‘small’ size architecture configuration with 225 layers and 1 ,1166,560 parameters to prioritize minimal inference time (i.e., speed of identifying potential spermatozoa during searching) over potentially greater recall from more parameters. Further image augmentations are applied by the YOLO model during the training process, including horizontal flipping, scaling, translation, and augmentations to hue, saturation and value. The training setup is restricted to a modest video random access memory of less than 8 GB, which limits the size of the YOLO model and training image resolution. In one arrangement, to maintain a high image resolution required to differentiate fine detail and the desired model size on this set-up, the training uses a small batch size of four images being trained in parallel. The YOLO model is trained for 300 training iterations or epochs with a learning rate of 0.01. The stochastic gradient descent optimizer is used with 0.937 momentum and 0.005 weight decay. The method 800 then proceeds from step 840 to step 850.

[0108] In step 850, the trained model is then used to make inferences on unseen, unlabelled images from the 15% of the images in the training dataset allocated for validation. The performance of the model is validated on images with a ground truth sperm number showing 85% precision and 78% recall after 300 epochs. The method 800 concludes at the conclusion of step 850.AH25(P0057851 PCT_46534242_1 DOCX)

[0109] As described hereinbefore in relation to the method 200, multiple neural models are trained for different settings of the microscope 110. This arrangement requires the method 800 to be performed at those settings to obtain the best neural model to use for those particular settings.

[0110] Figs. 6A to 6C and 7A to 7B show graphs comparing the performance of an embroylogist and the trained model executing the method 400. Fig. 6A shows that the embroylogist takes on average 36.10 seconds to locate sperm in a field of view, while the trained model takes on average 0.02 seconds to locate the sperm in the same field of view. Fig. 6B show a comparison of the recall of detecting sperm in a field of view between the embroylogist and the trained model. Fig. 6C shows a precision comparison of performance between the model and an embryologist . Fig. 7A shows the time taken on average to locate sperm per droplet. Fig. 7B shows the number of sperm located by the embroylogist and the trained model based on the number of droplets. As shown in these figures, the trained model outperforms the embroylogist in all metrics.Industrial Applicability

[0111] The arrangements described are applicable to the computer and data processing industries and particularly for the assisted human reproduction industries.

[0112] The foregoing describes only some embodiments of the present invention, and modifications and / or changes can be made thereto without departing from the scope and spirit of the invention, the embodiments being illustrative and not restrictive.

[0113] In the context of this specification, the word “comprising” means “including principally but not necessarily solely” or “having” or “including”, and not “consisting only of”. Variations of the word "comprising", such as “comprise” and “comprises” have correspondingly varied meanings.AH25(P0057851 PCT_46534242_1 DOCX)

Claims

CLAIMS:

1. A method of detecting sperm, comprising: receiving an image of a portion of a sample; detecting sperm in the image using a model trained for detecting sperm; and indicating the sperm detected in the image.

2. The method of claim 1, further comprising: placing a bounding box on the detected sperm.

3. The method of claim 1 or 2, wherein the image is captured by a camera of a microscope.

4. The method of claim 3, further comprising: transmitting control signals to the microscope to set a location, brightness, and focal length of the camera for capturing the image.

5. The method of claim 4, wherein the location and the focal length of the camera are adjusted to capture images of portions of the sample.

6. The method of claim 5, wherein the detected sperm is updated based on the sperm detected on the images with different focal lengths of the camera.

7. The method of any one of claims 1 to 5, wherein the sperm detection is performed by a neural model.

8. The method of claim 7, wherein the neural model is any one of a You Only Look Once (YOLO) model, region proposals, single shot multibox detector, and retina net, wherein the neural model is trained using a training dataset comprising images of samples and duplicates of the images of samples.

9. The method of claim 8, wherein the training dataset further comprises augmentation of the images of samples and the duplicates of the images of samples, wherein the augmentation comprises any one of flipping, blurring, darkening, contrasting, colouring, and magnification.

10. The method of any one of claims 7 to 9, wherein the neural model used to detect the sperm is selected from a plurality of neural models that are trained for different settings of the camera capturing the image.AH25(P0057851 PCT_46534242_1 DOCX)11. The method of any one of claims 1 to 10, further comprising generating a notification in response to detecting the sperm.

12. A system for detecting sperm, the system comprising: a computer system comprising: a processor; and memory in communication with the processor, the memory comprising a computer application program that is executable by the processor to perform a method comprising; receiving an image of a portion of a sample; detecting sperm in the image using a model trained for detecting sperm; and indicating the sperm detected in the image; and a microscope having a camera, wherein the microscope is in communication with the processor, the microscope being configured to capture an image of the portion of the sample and to transmit the captured image to the processor.

13. The system of claim 12, wherein the computer system comprises a video display, wherein the method further comprises placing a bounding box on the detected sperm for display on the video display.

14. The system of claim 12 or 13, wherein the method further comprises transmitting control signals to the microscope to set a location, brightness, and focal length of the camera for capturing the image.

15. The system of claim 14, wherein the location and the focal length of the camera are adjusted to capture images of portions of the sample.

16. The system of claim 15, wherein the detected sperm is updated based on the sperm detected on the images with different focal lengths of the camera.

17. The system of any one of claims 12 to 16, wherein the computer application program includes a neural model for detecting sperm.AH25(P0057851 PCT_46534242_1 DOCX)18. The system of claim 17, wherein the neural model is any one of a You Only Look Once (YOLO) model, region proposals, single shot multibox detector, faster R-CNN, Mask R-CNN, EfficientDet, Vision Transformer, and retina net, wherein the neural model is trained using a training dataset comprising images of samples and duplicates of the images of samples.

19. The system of claim 18, wherein the training dataset further comprises augmentation of the images of samples and the duplicates of the images of samples, wherein the augmentation comprises any one of flipping, blurring, darkening, contrasting, colouring, and magnification.

20. The system of any one of claims 17 to 19, wherein the neural model used to detect the sperm is selected from a plurality of neural models that are trained for different settings of the camera capturing the image.

21. The system of any one of claims 12 to 20, wherein the system generates a notification in response to detecting the sperm.AH25(P0057851 PCT_46534242_1 DOCX)

Images