Panoramic fundus image forming method and device, imaging device and storage medium

Abstract

The invention discloses a panoramic fundus image forming method and device, an imaging device and a storage medium, and the panoramic fundus image forming method comprises the following steps: S100, setting a fixed viewpoint motion parameter, guiding eyeballs of a collected person to move along with the movement of a fixed viewpoint, and collecting a fundus image sequence of the collected person in real time at a preset frequency; s200, iteratively splicing a panoramic fundus image in real time based on each frame of fundus image in the fundus image sequence; and S300, outputting the panoramic eye fundus image.

Description

Technical Field

[0001] The present disclosure generally relates to the technical field of fundus imaging, and more specifically, to a method for forming a panoramic fundus image, a forming device, an imaging device, and a storage medium. Background Art

[0002] Eyes are the human visual organs. Fundus images have important reference value in the clinical diagnosis and treatment of ophthalmology. At present, fundus images are usually obtained with the help of fundus angiography, professional digital cameras and special acquisition equipment to obtain high-resolution fundus images. Clinically, the fundus refers to the general term for the eyeball wall composed of the retina, choroid, sclera and the inner segment of the optic nerve, as well as the vitreous body closely related to it. The retinal blood vessels in the fundus are the only blood vessels that can be directly observed in the human body without trauma, and include the three major blood vessels in the human body: arteries, veins, and capillaries. The health of the human body can be reflected through the blood vessels. Therefore, fundus examination is not only an important means to examine vitreous, retinal, choroidal, and optic nerve diseases, but also an important means for certain cranial, cardiovascular, blood, endocrine and other systemic diseases. Its examination has a certain reference value in advance for the diagnosis of certain systemic diseases.

[0003] Due to the limitations of fundus cameras, the images obtained can only be partial images of the fundus. This means that in clinical diagnosis and treatment, ophthalmologists often need to compare and analyze two or more fundus retinal images taken at different times of the same patient. However, due to the differences in imaging angles and imaging environments, the accuracy and efficiency of comparisons based solely on naked-eye observation by doctors are low. There are two ways to solve this problem in the prior art. One is to increase the field of view of the device imaging, but this usually requires relatively expensive costs and is unrealistic for most hospitals. Another way is to splice multiple fundus images to meet clinical application requirements such as expanding the field of view, disease tracking, and auxiliary diagnosis. Therefore, fundus retinal image splicing is an important part of fundus retinal lesion detection and is of great significance to improving the accuracy of ophthalmologists' disease diagnosis. Summary of the invention

[0004] Some embodiments of the present disclosure provide a method for forming a panoramic fundus image, comprising the following steps:

[0005] S100: Setting the fixation point movement parameters, guiding the subject's eyeball to move along with the movement of the fixation point, and collecting the subject's fundus image sequence in real time at a predetermined frequency;

[0006] S200: Iteratively stitching a panoramic fundus image in real time based on each frame of the fundus image sequence; and

[0007] S300: Outputting the panoramic fundus image.

[0008] In some embodiments, the real-time iterative stitching of the panoramic fundus image based on each frame of the fundus image sequence comprises the following steps:

[0009] S210: constructing a panoramic fundus image area of ​​a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image;

[0010] S220: Iteratively stitching a panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus images.

[0011] In some embodiments, the steps of constructing a panoramic fundus image region of a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image include the following steps:

[0012] S211: constructing a panoramic fundus image area of ​​a predetermined size, wherein the panoramic fundus image area is a blank area;

[0013] S212: Quantitatively evaluate the image quality of the fundus image of the current frame, and determine whether the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold. If the image quality coefficient of the fundus image of the current frame is less than or equal to the image quality threshold, discard the fundus image of the current frame, obtain the next fundus image and update it as the fundus image of the current frame; if the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, determine the current frame image as the reference frame fundus image;

[0014] S213: placing the reference frame fundus image in the middle of the panoramic fundus image area to obtain an initialized panoramic fundus image.

[0015] In some embodiments, the iterative stitching of the panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus image comprises the following steps:

[0016] S221: Quantitatively evaluate the image quality of the fundus image of the current frame, determine whether the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, if the image quality coefficient of the fundus image of the current frame is less than or equal to the image quality threshold, discard the fundus image of the current frame, obtain the next frame of fundus image and update it as the fundus image of the current frame; if the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, use the image registration method to register the fundus image of the current frame with the fundus image of the reference frame, and determine the offset vector between the fundus image of the current frame and the fundus image of the reference frame;

[0017] S222: Determine whether the offset vector between the current frame fundus image and the reference frame fundus image is greater than the offset threshold. If the offset vector is less than or equal to the offset threshold, geometrically transform the current frame fundus image to the panoramic fundus image coordinate system, and superimpose and denoise it with the reference frame fundus image; if the offset vector is greater than the offset threshold, geometrically transform the current frame fundus image to the panoramic fundus image coordinate system, and splice and fuse it with the reference frame fundus image, and update the current frame fundus image to the reference frame fundus image.

[0018] In some embodiments, the registering the fundus image of the current frame with the fundus image of the reference frame by using an image registration method to determine the offset vector between the fundus image of the current frame and the fundus image of the reference frame includes:

[0019] An image registration method is used to align the current frame fundus image with the reference frame fundus image, and based on the image registration method, a translation vector of the current frame fundus image relative to the reference frame fundus image in the horizontal and vertical directions is determined, and based on the translation vector, a Euclidean distance is used to calculate the translation pixel distance as an offset vector between the current frame fundus image and the reference frame fundus image.

[0020] In some embodiments, setting the fixation point movement trajectory, guiding the subject's eyeball to move along with the movement of the fixation point, and collecting the subject's fundus image sequence in real time at a predetermined frequency comprises the following steps:

[0021] S110: Setting the motion parameters of the fixation point, wherein the motion parameters of the fixation point include: a running path trajectory, a stay time, and a motion speed of the fixation point;

[0022] S120: guiding the subject's eyeball to move along with the movement of the fixation point, using pupil tracking technology to track the subject's pupil center position in real time, and acquiring the subject's fundus image sequence in real time at a predetermined frequency based on the pupil center position.

[0023] In some embodiments, outputting the panoramic fundus image comprises:

[0024] Implementing the panoramic fundus image during the output stitching process; or

[0025] After the panoramic fundus image stitching is completed, the complete panoramic fundus image is output.

[0026] Some embodiments of the present disclosure provide a device for forming a panoramic fundus image, the device comprising:

[0027] The fundus image acquisition device is configured to set the fixation point movement parameters, guide the subject's eyeball to move along with the movement of the fixation point, and acquire the subject's fundus image sequence in real time at a predetermined frequency;

[0028] A panoramic fundus image stitching device, configured to iteratively stitch the panoramic fundus image in real time based on each frame of fundus image in the fundus image sequence; and

[0029] The panoramic fundus image output device is configured to output the panoramic fundus image.

[0030] Some embodiments of the present disclosure provide an imaging device for panoramic fundus images, including a processor and a memory, characterized in that the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the method steps described in the above embodiments are implemented.

[0031] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer program instructions, which implement the method steps described in the aforementioned embodiments when called and executed by a processor.

[0032] According to the technical solution of the embodiments of the present disclosure, based on a fixed-point motion path set in advance, the collector's eye movement is guided to collect a sequence of fundus images of the subject in real time, and the panoramic fundus image is iteratively stitched in real time based on the fundus image sequence and output for display, so that the panoramic fundus image of the subject is presented to the subject or medical staff in real time, thereby improving the diagnosis and treatment efficiency of medical staff and enhancing the experience of clinical users. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and other features of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0034] Figure 1 A flowchart of a method for forming a panoramic fundus image provided in some embodiments of the present disclosure;

[0035] Figure 2 for Figure 1 Specific flow chart of step 200;

[0036] Figure 3 for Figure 2 Specific flow chart of step 210;

[0037] Figure 4 for Figure 2 Specific flow chart of step 220;

[0038] Figure 5 for Figure 1 Specific flow chart of step 100;

[0039] Figure 6 A schematic diagram of the structure of a device for forming a panoramic fundus image provided in some embodiments of the present disclosure; and

[0040] Figure 7 An electronic structure diagram of an imaging device for panoramic fundus images provided in some embodiments of the present disclosure.

[0041] In the drawings, the same or similar structures are marked with the same or similar reference numerals. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0043] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0044] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0045] It should be understood that although the terms first, second, third, etc. may be used to describe structures in the disclosed embodiments, these structures should not be limited to these terms. These terms are only used to distinguish different structures. For example, without departing from the scope of the disclosed embodiments, a first component may also be referred to as a second component, and similarly, a second component may also be referred to as a first component.

[0046] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0048] In the related art, image configuration technology is used to stitch multiple fundus images, but the eyeballs of the subject are not regularly guided when taking the fundus images. The stitching of fundus images is a subsequent processing after the multiple fundus images are collected, which is not time-effective.

[0049] The present disclosure provides a method for forming a panoramic fundus image, comprising the following steps: S100: setting a fixation point motion parameter, guiding the subject's eyeball to move along with the movement of the fixation point, and acquiring a fundus image sequence of the subject in real time at a predetermined frequency; S200: iteratively stitching a panoramic fundus image based on each frame of the fundus image sequence in real time; and S300: outputting the panoramic fundus image.

[0050] In the present invention, based on a fixed-point motion path set in advance, the collector's eye movement is guided to collect a fundus image sequence of the subject in real time, and the panoramic fundus image is iteratively stitched in real time based on the fundus image sequence and output for display, so that the panoramic fundus image of the subject is presented to the subject or medical staff in real time, thereby improving the diagnosis and treatment efficiency of medical staff and enhancing the experience of clinical users.

[0051] The specific embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0052] Figure 1 Flow chart of a method for forming a panoramic fundus image provided by some embodiments of the present disclosure. Figure 1 As shown, some embodiments of the present disclosure provide a method for forming a panoramic fundus image, comprising the following steps:

[0053] S100: Setting the fixation point movement parameters, guiding the subject's eyeball to move along with the movement of the fixation point, and acquiring the subject's fundus image sequence in real time at a predetermined frequency.

[0054] Specifically, the fixation point refers to the target point that the eyes are looking at, which can be a light point that appears in the subject's field of vision. During the process of collecting ophthalmic images, the subject needs to adjust the position of the eyeball and focus the viewpoint on the fixation point. By changing the position of the fixation point, the subject's eyeball movement is guided to collect ophthalmic images at different angles or positions.

[0055] S200: Iteratively stitching a panoramic fundus image in real time based on each frame of the fundus image sequence.

[0056] Specifically, the fundus image of the current frame is acquired from the ophthalmic imaging module in real time, and it is determined whether the fundus image of the current frame has good image quality and whether it participates in image enhancement or image stitching according to the image registration method and threshold conditions, and the panoramic fundus image stitching map is initialized and iteratively updated.

[0057] S300: Outputting the panoramic fundus image.

[0058] Specifically, a display device is used to present a panoramic fundus image that has been stitched together or a panoramic fundus image that is in the process of being stitched together.

[0059] Figure 2 for Figure 1 The specific flow chart of step 200 is as follows: Figure 2 As shown, in some embodiments, the step S200: real-time iterative stitching of the panoramic fundus image based on each frame of the fundus image sequence includes the following steps:

[0060] S210: constructing a panoramic fundus image region of a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image.

[0061] Specifically, a panoramic fundus image mosaic image object is created according to a predetermined size, which is initially a blank image, a reference frame fundus image is determined, and the panoramic fundus image is initialized based on the reference frame fundus image.

[0062] S220: Iteratively stitching a panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus images.

[0063] Specifically, the image registration technology is used to iteratively stitch subsequent frame fundus images into the panoramic fundus image.

[0064] Figure 3 for Figure 2 The specific flow chart of step 210 is as follows: Figure 3 As shown, in some embodiments, the step S210: constructing a panoramic fundus image area of ​​a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image includes the following steps:

[0065] S211: constructing a panoramic fundus image area of ​​a predetermined size, where the panoramic fundus image area is a blank area.

[0066] Specifically, a panoramic fundus image area is created according to a predetermined size, and the panoramic fundus image area is a blank area where no image exists.

[0067] S212: Quantitatively evaluate the image quality of the current frame fundus image to determine whether the image quality coefficient of the current frame fundus image is greater than the image quality threshold. If the image quality coefficient of the current frame fundus image is less than or equal to the image quality threshold, discard the current frame fundus image and obtain the next frame fundus image to update it to the current frame fundus image. If the image quality coefficient of the current frame fundus image is greater than the image quality threshold, determine the current frame image as the reference frame fundus image.

[0068] Specifically, the acquired fundus images are processed according to the order of the fundus images in the fundus image sequence. For example, an automatic evaluation method is used to quantitatively evaluate the image quality of the current frame fundus image. The quantified value is called the image quality coefficient of the current frame fundus image. The automatic evaluation method includes, for example, the Laplacian variance method, the Brenner variance method, etc.

[0069] The image quality coefficient of the current frame fundus image is compared with a preset image quality threshold. If the image quality coefficient of the current frame fundus image is greater than the image quality threshold, the current frame image is determined as the reference frame fundus image. If the image quality coefficient of the current frame fundus image is less than or equal to the image quality threshold, the current frame fundus image is discarded, and the next frame fundus image is obtained and updated as the current frame fundus image. The image quality quantitative evaluation is performed again, and the image quality coefficient of the current frame fundus image obtained is compared with the preset image quality threshold until the reference frame fundus image is determined.

[0070] S213: placing the reference frame fundus image in the middle of the panoramic fundus image area to obtain an initialized panoramic fundus image.

[0071] Specifically, the reference frame fundus image is placed in the middle of the panoramic fundus image area, for example, the center of the reference frame fundus image is located in the center of the panoramic fundus image area, and the initialization of the panoramic fundus image is completed.

[0072] Figure 4 for Figure 2 The specific flow chart of step 220 is as follows: Figure 4 As shown, the step S220: iteratively stitching the panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus image includes the following steps:

[0073] S221: Quantitatively evaluate the image quality of the fundus image of the current frame to determine whether the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold. If the image quality coefficient of the fundus image of the current frame is less than or equal to the image quality threshold, discard the fundus image of the current frame and obtain the next frame to update it as the fundus image of the current frame. If the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, use the image registration method to register the fundus image of the current frame with the fundus image of the reference frame to determine the offset vector between the fundus image of the current frame and the fundus image of the reference frame.

[0074] Specifically, after completing the initialization of the panoramic fundus image, continue to process the acquired fundus images according to the order of the fundus images in the fundus image sequence, for example, use an automatic evaluation method to quantitatively evaluate the image quality of the current frame fundus image. The quantified value is called the image quality coefficient of the current frame fundus image. The automatic evaluation method includes, for example, the Laplacian variance method, the Brenner variance method, etc.

[0075] The image quality coefficient of the current frame fundus image is compared with the preset image quality threshold. If the image quality coefficient of the current frame fundus image is greater than the image quality threshold, the current frame fundus image is confirmed to be a valid fundus image, which can be subsequently stitched with the panoramic fundus image. The effective current frame fundus image is registered with the reference frame fundus image by an image registration method, such as homography registration of key points and descriptors, pixel offset field estimation method, etc., to determine the offset vector between the current frame fundus image and the reference frame fundus image. If the image quality coefficient of the current frame fundus image is less than or equal to the image quality threshold, the current frame fundus image is discarded, the next frame fundus image is obtained and updated as the current frame fundus image, and quantitative evaluation is continued until a valid current frame fundus image is obtained, and the valid current frame fundus image is registered with the reference frame fundus image to determine the offset vector between the current frame fundus image and the reference frame fundus image.

[0076] In some embodiments, the current frame fundus image is registered with the reference frame fundus image by using an image registration method, and determining the offset vector between the current frame fundus image and the reference frame fundus image includes:

[0077] An image registration method is used, such as homography registration based on key points and descriptors, and pixel offset field estimation method, to align the effective current frame fundus image with the reference frame fundus image, and based on the image registration method, the translation vector of the current frame fundus image relative to the reference frame fundus image in the horizontal and vertical directions is determined, such as the translation vector of the center of the current frame fundus image relative to the center of the reference frame fundus image in the horizontal and vertical directions in the panoramic fundus image coordinate system, and based on the translation vector, the Euclidean distance is used to calculate the translation pixel distance as the offset vector of the current frame fundus image and the reference frame fundus image, and the translation pixel distance represents the "offset degree" of the current frame fundus image and the reference frame fundus image.

[0078] S222: Determine whether the offset vector between the current frame fundus image and the reference frame fundus image is greater than the offset threshold. If the offset vector is less than or equal to the offset threshold, geometrically transform the current frame fundus image to the panoramic fundus image coordinate system, and superimpose and denoise it with the reference frame fundus image; if the offset vector is greater than the offset threshold, geometrically transform the current frame fundus image to the panoramic fundus image coordinate system, and splice and fuse it with the reference frame fundus image, and update the current frame fundus image to the reference frame fundus image.

[0079] Specifically, it is determined whether the translation pixel distance of the current frame fundus image relative to the reference frame fundus image is greater than a preset offset threshold. If the offset vector is less than or equal to the offset threshold, it is considered that the current frame fundus image and the reference frame fundus image are highly overlapped in image content, and the current frame fundus image can be used to enhance the image quality of the reference frame fundus image, that is, superimposed noise reduction. Specifically, according to the result of image registration, the current frame fundus image is geometrically transformed into the coordinate system of the reference frame fundus image, the average value of the pixels of the overlap of the two frames of fundus images is calculated, and the result is assigned to the reference frame fundus image, thereby enhancing the image quality of the reference frame fundus image.

[0080] If the offset vector is greater than the offset threshold, it is considered that the current frame fundus image has "moved away" from the reference frame fundus image, and the current frame fundus image can be used to expand the content of the panoramic mosaic image, and an image stitching operation should be performed. Specifically, according to the result of image registration, the current frame fundus image is geometrically transformed into the coordinate system of the panoramic mosaic image, and image stitching and fusion are performed with the reference frame fundus image, for example, feathering fusion, multi-band fusion, etc. are used to perform image stitching and fusion; after the fusion is completed, the current frame fundus image expands the visible range of the panoramic fundus image, and the current frame fundus image is defined as a new reference frame fundus image.

[0081] The above operation is performed on subsequent frames of fundus images in the fundus image sequence until the fundus image operation in all fundus image sequences is completed, thereby obtaining a panoramic fundus image.

[0082] Figure 5 for Figure 1 The specific flow chart of step 100 is as follows: Figure 5 As shown, the step S100: setting the fixation point movement trajectory, guiding the subject's eyeball to move along with the movement of the fixation point, and collecting the subject's fundus image sequence in real time at a predetermined frequency includes the following steps:

[0083] S110: Setting the fixation point movement parameters, wherein the fixation point movement parameters include: a running path trajectory, a stay time, and a movement speed of the fixation point.

[0084] Specifically, the relevant parameters of the fixation point are set according to actual needs, and the relevant parameters include but are not limited to: the running path trajectory, the dwell time, the moving speed (including the translation speed and the rotation angular velocity) and other motion-related parameters of the fixation point during the acquisition process.

[0085] S120: guiding the subject's eyeball to move along with the movement of the fixation point, using pupil tracking technology to track the subject's pupil center position in real time, and acquiring the subject's fundus image sequence in real time at a predetermined frequency based on the pupil center position.

[0086] Specifically, the fixation point position is initialized according to the preset fixation point motion trajectory, so that the subject should observe the fixation point in the visual field. The subject's eyeball is guided to move with the movement of the fixation point, and the imaging module is controlled to collect fundus images at a preset frame rate, and the images are transmitted to the image processing module in real time to obtain the subject's fundus image sequence; during the acquisition process, the pupil tracking method is used to track the subject's pupil center position in real time, and the position information is fed back to the imaging module for adjusting the scanning parameters to ensure the stability of the imaging quality. The scanning parameters include but are not limited to: sensor position, laser intensity, etc.

[0087] In some embodiments, the step S300: outputting the panoramic fundus image includes: outputting the panoramic fundus image in the stitching process; or outputting a complete panoramic fundus image after the stitching of the panoramic fundus images is completed.

[0088] For example, a display device can be used to present the panoramic fundus image in the stitching process in real time and enhance the clinical user experience. A display device can also be used to present the stitched panoramic fundus image in time, so that medical staff can view it in time and improve the diagnosis and treatment efficiency of medical staff.

[0089] Figure 6 A schematic diagram of a panoramic fundus image forming device provided in some embodiments of the present disclosure is shown in FIG. Figure 6As shown, some embodiments of the present disclosure further provide a panoramic fundus image forming device, wherein the panoramic fundus image forming device 100 comprises: a fundus image acquisition device 10 , a panoramic fundus image stitching device 20 and a panoramic fundus image output device 30 .

[0090] The fundus image acquisition device 10 is configured to set the fixation point movement parameters, guide the subject's eyeball to move along with the movement of the fixation point, and acquire the subject's fundus image sequence in real time at a predetermined frequency. The panoramic fundus image stitching device 20 is configured to iteratively stitch the panoramic fundus image in real time based on each frame of the fundus image sequence; the panoramic fundus image output device 30 is configured to output the panoramic fundus image.

[0091] Some embodiments of the present disclosure provide a non-transitory computer-readable storage medium storing computer program instructions, which, when called and executed by a processor, implement the steps of the method for forming a panoramic fundus image as described in any of the above embodiments.

[0092] Some embodiments of the present disclosure provide an imaging device for panoramic fundus images, including a processor and a memory, wherein the memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the method steps of any of the aforementioned embodiments are implemented.

[0093] Figure 7 The electronic structure diagram of the imaging device of the panoramic fundus image provided by some embodiments of the present disclosure is as follows: Figure 7 As shown, the imaging device of the panoramic fundus image may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In RAM 703, various programs and data required for the operation of the imaging device are also stored. The processing device 701, ROM 702, and RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0094] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the imaging device to communicate with other imaging devices wirelessly or by wire to exchange data. Although Figure 7An imaging device having various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0095] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as an imaging device software program. For example, an embodiment of the present disclosure includes an imaging device software program product, which includes a computer program carried on a readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 709, or installed from a storage device 708, or installed from a ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0096] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0097] The computer-readable medium may be included in the imaging device, or may exist independently without being incorporated into the imaging device.

[0098] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0099] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0100] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present disclosure.

Claims

1. A method for forming a panoramic fundus image, characterized in that: The following steps are involved: S100: Setting the fixation point movement parameters, guiding the subject's eyeball to move along with the movement of the fixation point, and collecting the subject's fundus image sequence in real time at a predetermined frequency; S200: Iteratively stitching a panoramic fundus image in real time based on each frame of the fundus image sequence; and S300: Outputting the panoramic fundus image.

2. The method for forming a panoramic fundus image according to claim 1, characterized in that: The method of iteratively stitching the panoramic fundus image in real time based on each frame of fundus image in the fundus image sequence comprises the following steps: S210: constructing a panoramic fundus image area of ​​a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image; S220: Iteratively stitching a panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus images.

3. The method for forming a panoramic fundus image according to claim 2, characterized in that: The steps of constructing a panoramic fundus image area of ​​a predetermined size, determining a reference frame fundus image, and initializing the panoramic fundus image include the following steps: S211: constructing a panoramic fundus image area of ​​a predetermined size, wherein the panoramic fundus image area is a blank area; S212: Quantitatively evaluate the image quality of the fundus image of the current frame, and determine whether the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold. If the image quality coefficient of the fundus image of the current frame is less than or equal to the image quality threshold, discard the fundus image of the current frame, obtain the next fundus image and update it as the fundus image of the current frame; if the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, determine the current frame image as the reference frame fundus image; S213: placing the reference frame fundus image in the middle of the panoramic fundus image area to obtain an initialized panoramic fundus image.

4. The method for forming a panoramic fundus image according to claim 2 or 3, characterized in that: The iterative stitching of the panoramic fundus image based on the reference frame fundus image and the subsequent frame fundus images comprises the following steps: S221: Quantitatively evaluate the image quality of the fundus image of the current frame, determine whether the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, if the image quality coefficient of the fundus image of the current frame is less than or equal to the image quality threshold, discard the fundus image of the current frame, obtain the next frame of fundus image and update it as the fundus image of the current frame; if the image quality coefficient of the fundus image of the current frame is greater than the image quality threshold, use the image registration method to register the fundus image of the current frame with the fundus image of the reference frame, and determine the offset vector between the fundus image of the current frame and the fundus image of the reference frame; S222: determining whether the offset vector between the current frame fundus image and the reference frame fundus image is greater than an offset threshold; if the offset vector is less than or equal to the offset threshold, geometrically transforming the current frame fundus image into a panoramic fundus image coordinate system, and superimposing and denoising the current frame fundus image with the reference frame fundus image; If the offset vector is greater than the offset threshold, the current frame fundus image is geometrically transformed into the panoramic fundus image coordinate system and spliced ​​and fused with the reference frame fundus image, and the current frame fundus image is updated to the reference frame fundus image.

5. The method for forming a panoramic fundus image according to claim 4, characterized in that: The method of using an image registration method to register the fundus image of the current frame with the fundus image of the reference frame, and determining the offset vector between the fundus image of the current frame and the fundus image of the reference frame includes: An image registration method is used to align the current frame fundus image with the reference frame fundus image, and based on the image registration method, a translation vector of the current frame fundus image relative to the reference frame fundus image in the horizontal and vertical directions is determined, and based on the translation vector, a Euclidean distance is used to calculate the translation pixel distance as an offset vector between the current frame fundus image and the reference frame fundus image.

6. The method for forming a panoramic fundus image according to any one of claims 1 to 3, characterized in that: The method of setting the fixation point movement trajectory, guiding the subject's eyeball to move along with the movement of the fixation point, and collecting the subject's fundus image sequence in real time at a predetermined frequency comprises the following steps: S110: Setting the motion parameters of the fixation point, wherein the motion parameters of the fixation point include: a running path trajectory, a stay time, and a motion speed of the fixation point; S120: guiding the subject's eyeball to move along with the movement of the fixation point, using pupil tracking technology to track the subject's pupil center position in real time, and acquiring the subject's fundus image sequence in real time at a predetermined frequency based on the pupil center position.

7. The method for forming a panoramic fundus image according to any one of claims 1 to 3, characterized in that: Outputting the panoramic fundus image comprises: Implementing the panoramic fundus image during the output stitching process; or After the panoramic fundus image stitching is completed, the complete panoramic fundus image is output.

8. A device for forming a panoramic fundus image, characterized in that: The forming device comprises: The fundus image acquisition device is configured to set the fixation point movement parameters, guide the subject's eyeball to move along with the movement of the fixation point, and acquire the subject's fundus image sequence in real time at a predetermined frequency; A panoramic fundus image stitching device, configured to iteratively stitch the panoramic fundus image in real time based on each frame of fundus image in the fundus image sequence; and The panoramic fundus image output device is configured to output the panoramic fundus image.

9. An imaging device for panoramic fundus images, comprising a processor and a memory, characterized in that: The memory stores computer program instructions that can be executed by the processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 7 are implemented.

10. A non-transitory computer-readable storage medium, characterized in that: Computer program instructions are stored, and when the computer program instructions are called and executed by a processor, the method steps according to any one of claims 1 to 7 are implemented.

Images