Information processing device, information processing method, and program

Abstract

The present disclosure discloses an information processing device, an information processing method, and a program. The information processing device includes: a storage unit configured to store information individually set for each type of imaging among a plurality of different types of imaging as a transmission setting for a plurality of imaging data obtained by the plurality of different types of imaging; and a transmission unit configured to transmit imaging data about a test subject based on the stored information, the imaging data being obtained by any one of the plurality of different types of imaging.

Description

Technical Field

[0001] The present invention relates to an information processing device, an information processing method and a program. Background Art

[0002] There has heretofore been known an ophthalmic imaging apparatus including a fundus camera and an optical coherence tomography (OCT), which is configured such that their two optical systems partially share an optical path (Patent Document 1).

[0003] Citation List

[0004] Patent Literature

[0005] PTL 1: Japanese Patent Laid-Open No. 2011-4978 Summary of the invention

[0006] Technical issues

[0007] Conventionally, an information processing apparatus to which an imaging device including a plurality of optical systems for performing different types of imaging is communicatively connected or an information processing apparatus inside such an imaging device can perform settings related to different types of imaging, settings related to imaging data, etc. An information processing apparatus to which at least one imaging device among a plurality of imaging devices for performing different types of imaging (such as a fundus camera and an optical coherence tomography (OCT)) is communicatively connected can also perform the aforementioned various types of settings, etc. Some such information processing apparatuses can only perform settings related to a plurality of different types of imaging data in general, and improved examiner convenience is desired.

[0008] One of the purposes of the disclosed technology is to realize separate settings related to different types of imaging data. The aforementioned purpose is not restrictive, and the following purpose can be regarded as another purpose of the present application: to provide operations and effects derived from the configuration discussed in the mode for realizing the present invention described below and which cannot be realized by conventional technology.

[0009] Solutions to technical problems

[0010] According to one aspect of the present disclosure, an information processing device includes: a storage unit configured to store information set individually for each type of imaging among a plurality of different types of imaging as a transmission setting for a plurality of imaging data obtained through the plurality of different types of imaging; and a transmission unit configured to transmit imaging data about a test subject based on the stored information, the imaging data being obtained through any type of imaging among the plurality of different types of imaging.

[0011] Beneficial Effects of the Invention

[0012] According to one of the disclosed techniques, settings related to different types of imaging data can be set individually. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] [ Figure 1 ] Figure 1 is a diagram schematically showing a configuration of an ophthalmologic imaging system according to a first exemplary embodiment.

[0014] [ Figure 2 ] Figure 2 : is a diagram showing an example of an automatic transfer setting screen of the ophthalmologic imaging system according to the first exemplary embodiment.

[0015] [ Figure 3 ] Figure 3 is a diagram showing an example of an imaging screen of the ophthalmologic imaging system according to the first exemplary embodiment.

[0016] [ Figure 4 ] Figure 4 is a diagram showing an example of an imaging screen of the ophthalmologic imaging system according to the first exemplary embodiment.

[0017] [ Figure 5 ] Figure 5 is a diagram showing an example of a flowchart of the operation of the ophthalmologic imaging system according to the first exemplary embodiment.

[0018] [ Fig. 6A ] Fig. 6A is a diagram illustrating an example of a report screen displayed on a display unit according to the second exemplary embodiment.

[0019] [ Figure 6B ] Figure 6B is a diagram illustrating an example of a report screen displayed on a display unit according to the second exemplary embodiment.

[0020] [ Figure 7 ] Figure 7 is a diagram illustrating an example of image quality enhancement processing according to the second exemplary embodiment.

[0021] [ Figure 8 ] Figure 8 is a diagram illustrating an example of a user interface according to the second exemplary embodiment.

[0022] [ Fig.9A ] Fig.9A is a diagram showing an example of a configuration of a neural network used as a machine learning engine according to a sixth modification.

[0023] [ Fig. 9B ] Fig. 9B is a diagram showing an example of a configuration of a neural network used as a machine learning engine according to a sixth modification.

[0024] [ Fig. 10A ] Fig. 10A is a diagram showing an example of a configuration of a neural network used as a machine learning engine according to a sixth modification.

[0025] [ Fig. 10B ] Fig. 10B is a diagram showing an example of a configuration of a neural network used as a machine learning engine according to a sixth modification. DETAILED DESCRIPTION

[0026] The exemplary embodiments for implementing the present invention will be described in detail below with reference to the accompanying drawings. Note that the size, material, shape, relative position, etc. of the components described in the following exemplary embodiments are freely selectable and can be modified based on the configuration or various conditions of the device to which the present invention is applied. In the accompanying drawings, the same reference numerals are used in different drawings to represent the same or functionally similar elements. In the accompanying drawings, some components, members, and processes that are not important in the description may be omitted.

[0027] (First Exemplary Embodiment)

[0028] Will refer to Figures 1 to 5 The ophthalmic imaging system 100 according to the present exemplary embodiment is described as an example of an information processing device. The ophthalmic imaging system 100 according to the present exemplary embodiment can convert the data format of imaging data into a preset data format depending on the ophthalmic imaging device that captures the image, and automatically transmit the data. The following description will be given by using a fundus camera and an optical coherence tomography (OCT) as examples of ophthalmic devices to be processed.

[0029] (System Configuration)

[0030] Will refer to Figure 1 Describe the system configuration. Figure 1The configuration of the ophthalmic imaging system 100 is schematically shown. The ophthalmic imaging system 100 includes an imaging data acquisition unit 101, an imaging data storage unit 102, a display control unit 103, an operation unit 104, an automatic transmission information storage unit 105, and an automatic transmission execution unit 106. For example, as employed in the present exemplary embodiment and the like, "transmission" may refer to a case where an imaging signal (e.g., an OCT interference signal) from an imaging device is simply transmitted as imaging data. As employed in the present exemplary embodiment and the like, "transmission" may also refer to a case where imaging data processed as an image generated from an imaging signal is transmitted. As employed in the present exemplary embodiment and the like, "transmission" may also refer to a case where data processed as a report image corresponding to a report screen including an image generated from an imaging signal is transmitted as imaging data. Examples of images generated from imaging signals may include: a frontal image (en-face image) and at least one tomographic image (B-scan image) acquired by using at least a portion of a depth range of a plurality of tomographic imaging image data (three-dimensional tomographic imaging image data, volume data) acquired at different positions. The information processing device may be configured to be able to set the depth range based on an instruction of an examiner. Here, the information processing device may be configured to set the depth range by changing the position of the layer boundary based on the inspector's instruction, the layer boundary being obtained by segmentation processing of the tomographic imaging image on the tomographic imaging image. As adopted in the present exemplary embodiment and the like, "automatic transmission" does not mean that the inspector's instruction will not be used as a trigger to start transmitting imaging data at all. For example, if the transmission setting of imaging is automatically set, it is desired that another instruction given by the inspector that is not originally intended to start transmitting imaging data (for example, an end-of-examination instruction) is also used as an instruction to start transmitting imaging data. In other words, the "automatic transmission" adopted in the present exemplary embodiment and the like may refer to any situation in which another instruction that is not directly intended to start transmitting imaging data is configured to also be used as an instruction to start transmitting imaging data.

[0031] The imaging data acquisition unit 101 can receive the captured imaging data from at least two or more ophthalmic imaging devices 110. In the present exemplary embodiment, the imaging data acquisition unit 101 can receive the fundus image and the retinal tomographic imaging data from the fundus camera and the optical coherence tomography imaging device, respectively. Here, although a plurality of imaging devices for performing different types of imaging can be communicatively connected to the information processing device at the same time, if at least one is connected, the present exemplary embodiment and the like are also applicable. More specifically, the information processing device is configured so that when any one of the imaging devices is communicatively connected to the information processing device, transmission settings for a plurality of imaging data from a plurality of imaging devices can be made separately. The information processing device according to the present exemplary embodiment may be an information processing device to which the imaging device is communicatively connected, the imaging device including a plurality of optical systems for performing different types of imaging, or may be an information processing device inside such an imaging device. The information processing device according to the present exemplary embodiment may be, for example, a personal computer. A desktop personal computer (PC), a laptop PC, or a tablet PC (portable information terminal) may be used.

[0032] The imaging data storage unit 102 receives and records the imaging data acquired by the imaging data acquisition unit 101. Here, additional information on the imaging data is additionally registered, including: patient information such as the name, date of birth, gender, patient identifier (ID), and race data of the patient (subject), examination information such as the examination date and time, reception number, and examination identification ID, and imaging information such as imaging time, imaging mode name, imaging parameters, device name, and imaging determination.

[0033] The display control unit 103 displays an imaging operation screen of the ophthalmic imaging device 110 when acquiring imaging data, and displays a confirmation screen for displaying imaging data acquired by the imaging data acquisition unit 101 and recorded in the imaging data storage unit 102, on a monitor not shown as an example of a display unit.

[0034] The operating unit 104 can operate the imaging of the ophthalmic imaging device 110 on the imaging operation screen, and select the imaging success / failure determination for the imaging data displayed on the imaging result confirmation screen via the mouse and keyboard. For the imaging success / failure determination of the imaging data, for example, the imaging determination information can be input by the operator confirming the imaging data displayed on the imaging result confirmation screen and pressing the success or failure button displayed on the screen. In addition, automatic transmission (automatic transmission) can be started via the operating unit 104. For example, the transition from the imaging operation screen to other screens can be used as an automatic transmission start trigger. Pressing the inspection completion button displayed on the imaging operation screen can be used as an automatic transmission start trigger. The display unit can be a touch panel display, in which case the display unit also serves as the operating unit 104.

[0035] The automatic transmission information storage unit 105 stores settings prepared in advance to automatically transmit imaging data. The stored settings include an ophthalmologic apparatus as an automatic transmission target, an automatic transmission destination, and a data format of imaging data to be transmitted.

[0036] The automatic transmission execution unit 106 receives an automatic transmission start trigger instructed from the operation unit 104 via the display control unit 103, and transmits the imaging data to the transmission data storage system 120 serving as the automatic transmission destination based on the automatic transmission information acquired from the automatic transmission information storage unit 105. Here, the automatic transmission execution unit 106 confirms the imaging data of which imaging device the automatic transmission information is for, converts the data format of the corresponding imaging data into the data format specified by the automatic transmission information, and transmits the converted data to the transmission data storage system 120.

[0037] (Automatic Transmission Information According to the Present Exemplary Embodiment)

[0038] Next, we will refer to Figure 2 The automatic transfer information stored in the automatic transfer information storage unit 105 is described. Figure 2 An example of a screen for setting automatic transmission (automatic transmission) content according to the present exemplary embodiment is shown. This screen is a screen for making automatic transmission settings. In a case such as when the setting is different from one imaging device to another imaging device and automatic transmission to multiple transmission destinations (transmission destinations) is intended, multiple automatic transmission settings can be registered in the automatic transmission information storage unit 105 by making corresponding settings respectively.

[0039] The transmission setting screen 200 includes a common setting area 210, an OCT examination setting area 220, and a fundus examination setting area 230. The OCT examination setting area 220 and the fundus examination setting area 230 are examples of individual setting areas.

[0040] The common setting area 210 is an area for setting transmission setting (transmission setting) items common to the OCT examination and the fundus examination, and includes a transmission content setting 211 , a transmission type setting 212 , a transmission destination setting 213 , and anonymization setting 214 .

[0041] The transmission content setting 211 (transmission content setting) can select "image" or "report" as the data content to be transmitted to the transmission destination (transmission destination). If "image" is selected as the transmission content setting, the image of the imaging data obtained by imaging the eye of the subject to be examined by the ophthalmic imaging device 110 is set to be transmitted. For example, in the case where the imaging data is obtained by capturing an image of the eye to be examined by a fundus camera, the fundus camera image is transmitted. In the case where the imaging data is obtained by capturing an image of the eye to be examined by OCT, at least one tomographic imaging image (B-scan image) is transmitted (transmitted). Since OCT imaging can obtain multiple tomographic imaging images through a single imaging operation, all multiple tomographic imaging images can be set to be transmitted. In some cases, OCT imaging can also capture a scanning laser ophthalmoscope (SLO) or other retinal frontal images at the same time. Then, the retinal frontal image can be set to be transmitted together. If OCT captures a three-dimensional image of the retina, the retinal frontal image, which is a frontal image reconstructed from the OCT imaging data, can be set to be transmitted together. In addition, an image in which a tomographic imaging image and a retinal front image are juxtaposed and an imaging position of the tomographic imaging image is indicated on the retinal front image can be set to be transmitted. If "report" is selected as the transmission content setting, a report image in which imaging data and related data are arranged in a specific layout (such as a report image in which multiple imaging data are displayed side by side or a report image in which multiple analysis information about the imaging data are displayed side by side) is set to be transmitted. For example, in the case of a fundus camera, a report image in which multiple captured images are arranged in a matrix is ​​transmitted. In the case of OCT, in addition to the tomographic imaging image and the retinal front image, a report image showing a map image can also be transmitted, which indicates the retinal thickness at various positions on the retinal front image in color gradation. The report image may include patient information, examination information, and imaging information registered in the imaging data storage unit 102 as additional information about the imaging data. The options for the transmission content (transmission content) are not limited to two-dimensional images such as various two-dimensional medical images and report images. For example, OCT three-dimensional structure data may be transmitted.

[0042] The transmission type setting 212 (transmission type setting) can select the data transmission method and image format of the content set by the transmission content setting 211. For example, if the transmission destination (transmission destination) is a medical digital imaging and communications (DICOM) storage server, DICOM communication is selected. If the transmission destination (transmission destination) is a storage such as a hard disk or a network attached storage (NAS), a file storage is selected. In the case of a file storage, the image format for storage is selected from bitmap, joint imaging experts group (JPEG) and DICOM. In other words, the content of the imaging data to be transmitted (attachment information, etc.) varies depending on the transmission type and storage format.

[0043] The transfer destination setting 213 (transmission destination setting) can set the transfer destination of the imaging data. The method for setting the transfer destination differs depending on whether the data transfer method in the transfer type setting 212 is DICOM communication or file storage, and the input items in the screen change based on the selection. If DICOM communication is selected, the host name, port number, and server application entity (AE) title required for communication with the transfer destination DICOM storage server can be entered. In this case, this screen can include a function to confirm whether communication with the input transfer destination is possible. If file storage is selected, the path of the storage location can be entered.

[0044] The anonymization setting 214 can select whether to anonymize personal information included in the data to be transmitted. If anonymization is set, personal information such as a patient name included in a report image, DICOM or JPEG tag information and / or a file name is set to be anonymous and transmitted. The anonymization setting may refer to a predetermined method for anonymization. An anonymization setting screen may be provided to make fine settings for the anonymization method, or an anonymization setting may be directly provided on the transmission setting screen.

[0045] The OCT examination setting area 220 is an area for setting items to be applied to the transmission of imaging data captured by OCT, and includes an OCT examination image size setting 221 and an OCT examination automatic transmission setting 222 .

[0046] The OCT examination image size setting 221 is an item to be enabled when the transmission content setting 211 is image, and the image size of the tomographic image to be transmitted can be set. Select Original size to transmit the tomographic image in the same size as the imaging data captured by OCT. Select Resize to display size to transmit the tomographic image in the size at which the ophthalmic imaging system 100 displays the tomographic image on the monitor.

[0047] The OCT examination automatic transmission setting 222 can select automatic transmission for OCT examination. For example, if automatic transmission is selected, the automatic transmission is set to be enabled. When the automatic transmission is performed, when the automatic transmission execution unit 106 receives an automatic transmission start trigger, if the ophthalmic imaging device that captures the imaging data is OCT, the automatic transmission execution unit 106 automatically transmits the content set on this transmission setting screen 200.

[0048] The fundus examination setting area 230 is an area for setting items to be applied to the transmission of imaging data captured by the fundus camera, and includes a fundus examination imaging size setting 231 and a fundus examination automatic transmission setting 232 .

[0049] If the Transmit Content Setting 211 is Image, the Fundus Examination Imaging Size Setting 231 is an item to be enabled and can set the image size of the fundus camera image to be transmitted. Select Original Size to transmit the fundus camera image at the same size as when captured. To transmit fundus camera images of different sizes, select an item for a specific width. When transmitting an image with a width less than or equal to the selected size, the image is set to be transmitted at its original size.

[0050] The fundus examination automatic transmission setting 232 can select automatic transmission (automatic transmission) for fundus examination. For example, if automatic transmission is selected, automatic transmission is set to be enabled. In the case of automatic transmission, when the automatic transmission execution unit 106 receives an automatic transmission start trigger, if the ophthalmic imaging device that captures imaging data is a fundus camera, the automatic transmission execution unit 106 automatically transmits the content set on this transmission setting screen 200.

[0051] (Imaging Screen According to the Present Exemplary Embodiment)

[0052] Figure 3 and Figure 4 An example of a screen for performing an examination using the ophthalmologic imaging apparatus displayed by the display control unit 103 is shown. The present exemplary embodiment relates to a screen example in which a transition from a screen for capturing an image using the ophthalmologic imaging apparatus 110 to another screen is used as an automatic transfer start trigger.

[0053] The ophthalmic imaging system screen 300 can display a plurality of screens, and the screens can be switched in a tab manner. Figure 3 An example of a screen for executing OCT imaging that is displayed when the OCT imaging tab 301 is selected is shown. Figure 4An example of a screen for performing fundus camera imaging displayed when the fundus imaging tab 402 is selected is shown. If the OCT imaging tab 301 is selected, an OCT imaging screen 310 is displayed in the tab, where OCT imaging can be performed and imaging results can be displayed. Here, Figure 3 A preview screen is shown, in which a moving image of the anterior eye portion is displayed in the upper left display area, an SLO moving image of the fundus is displayed in the lower left display area, and an OCT tomographic imaging moving image is displayed in the right display area. Here, if various optical adjustments such as alignment, focus, coherence gate, etc. are made and OCT imaging is performed on the preview screen, the information processing device can be configured to switch to the display of an imaging confirmation screen not shown. If the OCT imaging on the imaging confirmation screen is OK, the information processing device can be configured to switch to the display of the preview screen. If the fundus imaging tab 302 is selected, the fundus camera imaging screen 410 is displayed in the tab, in which the image captured by the fundus camera can be displayed.

[0054] Tabs other than the imaging screen include a report tab 303 that displays a report screen, in which imaging data about the imaged patient is displayed, and a patient tab 304 that displays a patient screen for creating and selecting a patient's record to start an examination of another patient or display imaging data about another patient. Here, the report screen can be configured to switch between various display screens, including a display screen for follow-up and a three-dimensional volume rendering display screen. The report screen can be configured so that one of the aforementioned various display screens can be set as its initial display screen. The report screen can be configured so that not only the initial display screen can be set as its initial display, but also the presence or absence of image quality enhancement processing, the presence or absence of analysis result display, the depth range for generating a frontal image, etc. can be set as its initial display. If "report" is selected as the aforementioned transmission content, a report image generated based on the content set as the initial display of the report screen can be transmitted. The report screen can be a display screen used in use cases such as performing OCT imaging after fundus imaging, and displays a fundus image, an OCT image, etc. together. For example, assuming that information in which transmission of a report image is set to be enabled is stored as a transmission setting, and as a setting regarding initial display of a report image, image quality enhancement processing is enabled, in this case, the transmission unit can transmit a report image corresponding to a report screen displaying a medical image obtained by image quality enhancement processing as imaging data.

[0055] In addition, there is a logout button 305 for logging out to end the use of the ophthalmic imaging system and displaying the login screen. If a tab other than the imaging screen or the logout button 305 is selected, automatic transmission is started. Since automatic transmission is not performed when the OCT imaging tab 301 is changed to the fundus imaging tab 302 or when the fundus imaging tab 302 is changed to the OCT imaging tab 301, the operation here is to automatically transmit both the imaging data of the fundus imaging and the imaging data of the OCT imaging in an integrated manner after performing both fundus imaging and OCT imaging. Not performing automatic transmission between imaging tabs can prevent the imaging operation of the operator from being hindered by the automatic transmission process. For example, in the case where the automatic transmission process is performed in parallel with the screen operation, the imaging operation can be prevented from failing due to the load of the automatic transmission process. In the case where the automatic transmission process is not performed in parallel with the screen operation but is completed before the next imaging screen is displayed, it can be prevented from letting the patient wait due to the automatic transmission process during a series of imaging operations on the patient. Note that if the automatic transmission processing is performed in parallel with the screen operation and the system has sufficient performance, the transition from the OCT imaging tab 301 to the fundus imaging tab 302 and the transition from the fundus imaging tab 302 to the OCT imaging tab 301 can also be processed as an automatic transmission start trigger. In this case, the imaging data of a single ophthalmic imaging device is always automatically transmitted. Only the patient tab 304 and the logout button 305 can be processed as automatic transmission start triggers, and the transition to the report screen is not processed as an automatic transmission start trigger, so the automatic transmission is performed in units of transitions from one patient to another. In this case, the information added and edited in the report screen can also be processed as information to be added at the time of automatic transmission.

[0056] (Automatic Transmission Processing Flow According to the Present Exemplary Embodiment)

[0057] Next, we will refer to Figure 5 The flow of the automatic transmission process according to the present exemplary embodiment is described. Figure 5 is an operation flowchart of the automatic transmission process according to the present exemplary embodiment.

[0058] The ophthalmic imaging device as the automatic transmission target may vary depending on the transmission setting. Therefore, when performing automatic transmission, the automatic transmission execution unit 106 confirms the automatic transmission setting in the automatic transmission information storage unit 105 and the ophthalmic imaging device 110 as the automatic transmission target stored in the imaging data storage unit 102, and performs automatic transmission to the transmission data storage system 120 only when the ophthalmic imaging device 110 is the automatic transmission target.

[0059] Specifically, in step S500, the operator performs imaging using the ophthalmic imaging apparatus 110, and the imaging data acquisition unit 101 acquires imaging data from the imaging apparatus 110. Here, the imaging data acquisition unit displays the imaging data via the display control unit 103, and the operator inputs whether the imaging is successful or failed.

[0060] In step S501, the imaging data acquisition unit 101 stores the imaging data acquired in step S500 and the result of whether the imaging is successful or failed input by the operator together with additional information in the imaging data storage unit 102. If imaging failure is input, the imaging data may or may not be stored in a location other than the imaging data storage unit 102.

[0061] In step S502, if the operator continues the examination, imaging is performed and the process returns to step S500. If the operator selects (presses) a tab other than the imaging tab (such as the OCT imaging tab 301 and the fundus imaging tab 302) on the ophthalmic imaging system screen 300, the imaging screen is switched to other display screens. Here, the examination is determined to have ended, and the automatic transmission start trigger is transmitted to the automatic transmission execution unit 106. For example, it is assumed that the fundus imaging has ended and the OCT imaging tab is pressed. The processing during the OCT preview (for example, the processing of the moving image intended for the anterior eye part, the SLO moving image of the fundus, the OCT tomographic imaging moving image, and various optical adjustments) is high in load. If the fundus imaging data is transmitted during the OCT preview, the processing during the OCT preview may therefore fail. Therefore, the information processing device may be configured not to perform automatic transmission not only when the current imaging tab is selected but also when other imaging tabs are selected. However, in the present exemplary embodiment and the like, the information processing device may be configured to perform automatic transmission when an imaging tab other than the currently selected imaging tab is selected. The information processing device can also be configured to switch the display screen to the login screen, and determine that the examination has ended if the logout button 305 is selected while the imaging screen is displayed. In this case, the selection of the logout button 305 can be used as an automatic transmission start trigger.

[0062] In step S503, the automatic transmission execution unit 106 reads the transmission settings one by one from the automatic transmission information storage unit 105, and determines whether there is an automatic transmission setting. If there is an automatic transmission setting, the process proceeds to step S504. If there is no automatic transmission setting, the automatic transmission execution unit 106 ends the automatic transmission process. If the information processing device is configured to be able to register only one transmission setting, step S503 and subsequent steps are not indispensable. As described above, multiple modes of transmission settings can be made registrable. In this case, if multiple modes of transmission settings are registered, in step S503, information (data) corresponding to the settings of the multiple modes of transmission settings can be transmitted in sequence. For example, even if the first transmission setting and the second transmission setting contradict each other, data about these settings can be transmitted in sequence.

[0063] In step S504, the automatic transmission execution unit 106 confirms the ophthalmic imaging device as the automatic transmission target in the automatic transmission setting confirmed in step S503 and the ophthalmic imaging device that has captured the imaging data stored in step S501, and determines whether the imaging data is included in the imaging data. The automatic transmission execution unit 106 confirms the settings of the OCT examination automatic transmission setting 222 and the fundus examination automatic transmission setting 232 in the transmission setting screen 200 for the ophthalmic imaging device as the automatic transmission target. If the imaging data includes the imaging data captured by the ophthalmic imaging device as the automatic transmission target, the processing proceeds to step S505 to enter the automatic transmission processing. If there is no imaging data captured by the ophthalmic imaging device as the automatic transmission target, the processing returns to step S503 to check whether there is a next automatic transmission setting.

[0064] In step S505 , the automatic transfer execution unit 106 sequentially reads the imaging data of the ophthalmologic imaging apparatus as the automatic transfer target from the imaging data stored in the imaging data storage unit 102 .

[0065] In step S506, the automatic transmission execution unit 106 performs data conversion on the imaging data read in step S505 based on the transmission setting. For example, if the imaging data is imaging data of a fundus camera, and an image is selected in the transmission content setting 211, and 1600 pixels (width) are selected in the image size setting 231, the automatic transmission execution unit 106 converts the image captured by the fundus camera into image information having a width of 1600 pixels. If the transmission type setting 212 is JPEG file storage, the automatic transmission execution unit 106 also performs data conversion to the JPEG format, and adds patient information, examination information, and imaging information to the JPEG tag. Here, if the anonymization setting 214 is set, the automatic transmission execution unit 106 anonymizes the personal information in the information included in the JPEG tag and adds the result.

[0066] In step S507, the automatic transmission execution unit 106 transmits the data converted in step S506 to the transmission destination set in the transmission destination setting 213. If file storage is selected in the transmission type setting 212, the automatic transmission execution unit 106 stores the file in the specified path. If DICOM communication is selected, the automatic transmission execution unit 106 transmits the data to the transmission data storage system as the transmission destination.

[0067] In step S508, the automatic transmission execution unit 106 confirms the result of the data transmission executed in step S507. If the transmission is completed normally, the process proceeds to step S510. If the transmission fails, the process proceeds to step S509 to execute retransmission processing.

[0068] In step S509, the automatic transmission execution unit 106 sets the data record for which transmission failed as a retransmission target. For example, if the storage location does not have sufficient capacity, or if the storage location or the communication destination cannot be accessed due to a network failure, the automatic transmission execution unit 106 registers a setting for automatically performing retransmission at the next login or after a certain time has passed. Retransmission may not be performed automatically but may be performed manually by an operator.

[0069] In step S510, the automatic transmission execution unit 106 confirms whether there is no other imaging data as a candidate for the automatic transmission setting confirmed in step S503. If there is such imaging data, the process proceeds to step S505 to automatically transmit the next imaging data. If there is no such imaging data, the process proceeds to step S503 to confirm whether there is other automatic transmission setting. Here, if all automatic transmissions have been completed in step S503, the automatic transmission ends. Here, the automatic transmission execution unit 106 can notify the result of the automatic transmission. As the notification content, it is notified whether all imaging data that are the targets of all automatic transmission settings have been successfully automatically transmitted. The automatic transmission execution unit 106 can notify the operator of information about the imaging data registered for retransmission in step S509.

[0070] In the processing of step S503 and subsequent steps in this automatic transmission processing flow, the automatic transmission processing can be performed from beginning to end separately from the screen display of the display control unit 103 and in parallel therewith, and the operator can perform screen operations. Alternatively, a transmission in progress message can be displayed and presented on the screen so that no screen operation is made before the automatic transmission processing is completed. If the automatic transmission processing is performed in parallel with the screen display, a notification of the automatic transmission result can be made after the automatic transmission processing is completed by preparing a result display area on the screen and displaying the result at the end of the automatic transmission. The details of the automatic transmission result can be displayed by the user then selecting the result display area.

[0071] According to the present exemplary embodiment described above, when automatically transmitting the captured imaging data, the ophthalmic imaging system 100 can perform automatic transmission while modifying the transmission content based on the type of ophthalmic imaging device that captured the imaging data. This is very advantageous because only the imaging data required by the ophthalmic imaging device is appropriately transmitted to the automatic transmission destination.

[0072] In the present exemplary embodiment, when the imaging of the ophthalmic imaging device is completed and the screen is changed to a different screen, automatic transmission is started. However, automatic transmission can be started at each imaging operation. In this case, after the imaging data is stored in step S501, the process proceeds to step S503 to perform automatic transmission. In the case of automatically transmitting a report image in which multiple imaging data are required, the operation here is to perform automatic transmission when it is determined that all required imaging data have been obtained. In the present exemplary embodiment, the imaging data obtained from each of the two devices (fundus camera and OCT) is automatically transmitted. However, this is not restrictive. If a single device has the functions of multiple ophthalmic imaging devices such as fundus camera and OCT, it is possible to perform automatic transmission appropriately on each ophthalmic imaging device function by confirming which ophthalmic imaging device function each imaging data is obtained by and automatically transmitting the imaging data obtained by the function of the ophthalmic imaging device as the automatic transmission target. In addition, in the present exemplary embodiment, automatic transmission is started in response to an automatic transmission start trigger after performing imaging. However, this is not restrictive. A button for integrated transmission may be prepared on a screen other than the imaging screen, and transmission may be started at an arbitrary timing selected by the operator, and transmission may be performed depending on the ophthalmic imaging device that is the target of automatic transmission. In this case, information on automatic transmission settings and imaging data to be automatically transmitted may be presented to the operator. The operator may be allowed to change the target automatic transmission settings and the range of imaging data, and start automatic transmission.

[0073] (Second exemplary embodiment)

[0074] The information processing device according to the present exemplary embodiment includes an image quality enhancement unit (not shown) for applying an image quality enhancement process based on machine learning as an image quality enhancement means for enhancing the image quality of motion contrast data. Here, the image quality enhancement unit of the information processing device generates a motion contrast image with high image quality (low noise and high contrast) equivalent to the image quality of the motion contrast image generated from a large number of tomographic images by inputting a low image quality motion contrast image generated from a small number of tomographic images into a machine learning model. As used herein, a machine learning model refers to a function generated by performing machine learning using training data, the training data including paired input data and output data (true value data), the input data being a low image quality image assumed to be a processing target and obtained under predetermined imaging conditions, and the output data being a high image quality image corresponding to the input data. The predetermined imaging conditions include an imaging area, an imaging method, an imaging viewing angle, and an image size.

[0075] For example, a low image quality motion contrast image is obtained in the following manner. The operator first operates the operation unit 104 to press the imaging start (capture) button in the imaging screen (preview screen), thereby starting optical coherence tomography angiography (OCTA) imaging under the set imaging conditions in response to the operator's instruction. Here, the control unit (not shown) of the information processing device instructs the optical coherence tomography (OCT) to perform OCTA imaging based on the settings specified by the operator, and obtains an OCT tomographic imaging image supported by the OCT. The OCT also obtains an SLO image by using an SLO optical system on an optical path separated by a dichroic mirror as an example of a wavelength separation member, and performs tracking processing based on the SLO motion image. Here, the imaging conditions are set, for example, by: 1) registering a macular disease inspection set, 2) selecting an OCTA scanning mode, and 3) setting the following imaging parameters, etc. Examples of the set imaging parameters include: 3-1) scanning mode: 300A-scan × 300B-scan, 3-2) scanning area size: 3×3 mm, and 3-3) main scanning direction: horizontal direction. Examples of imaging parameters that can be set also include 3-4) scanning interval: 0.01 mm, 3-5) fixation light position: macula (fovea), 3-6) number of B-scans per cluster: 4, 3-7) coherence gate position: vitreous side, and 3-8) predetermined display report type: monocular examination report. The imaging data acquisition unit 101 generates a motion contrast image (motion contrast data) based on the acquired OCT tomographic imaging image. Here, after the motion contrast image is generated, a correction unit not shown may perform processing for reducing projection artifacts that appear on the motion contrast image. The display control unit 103 then displays the generated tomographic imaging image, three-dimensional motion contrast image, motion contrast frontal image, information about imaging conditions, and the like on a display unit (not shown). Here, in the present exemplary embodiment, the image quality enhancement unit is activated by the user pressing the Fig. 6A 911 (an example of an image quality enhancement button) displayed in the upper right portion of the report screen in , to perform image quality enhancement processing on the motion contrast image. In other words, the image quality enhancement button is a button for giving an instruction to perform image quality enhancement processing. It will be understood that the image quality enhancement button may be a button for giving an instruction to display a high image quality image (generated before pressing the image quality enhancement button).

[0076] In the present exemplary embodiment, the input data used as training data is a low-image-quality motion-contrast image generated from a single cluster having a small number of tomographic images. The output data (true value data) used as training data is a high-image-quality motion-contrast image obtained by taking an additive average of a plurality of aligned motion-contrast data. Note that the output data to be used as training data is not limited thereto. For example, a high-image-quality motion-contrast image generated from a single cluster including a large number of tomographic images may be used. The output data to be used as training data may be a high-image-quality motion-contrast image obtained by reducing a motion-contrast image having a higher resolution (higher magnification) than the input image to the same resolution (same magnification) as the input image. The paired input and output image pairs used to train the machine learning model are not limited to the foregoing, and any combination of conventional images may be used. For example, when training the machine learning model, an image obtained by adding a first noise component to a motion-contrast image obtained by the ophthalmic imaging system 100 or other device may be used as an input image, and an image obtained by adding a second noise component (different from the first noise component) to a motion-contrast image (obtained by the ophthalmic imaging system 100 or other device) may be used as an output image. In other words, any image quality enhancement unit can enhance the image quality of motion contrast data input as an input image using a training model for image quality enhancement obtained by training using training data including motion contrast data of the fundus.

[0077] Figure 7The configuration of the machine learning model in the image quality enhancement unit according to the present exemplary embodiment is shown. The machine learning model is a convolutional neural network (CNN) and includes multiple layers responsible for processing a set of input values ​​and outputting the results. The multiple types of layers included in the aforementioned configuration include convolution layers, downsampling layers, upsampling layers, and merging layers. The convolution layer is a layer for performing convolution processing on a set of input values ​​based on parameters such as the kernel size of the set filter, the number of filters, the stride value, and the dilation value. The number of dimensions of the kernel size of the filter can be changed based on the number of dimensions of the input image. The downsampling layer is a layer for performing processing by extracting or combining the set of input values ​​to make the number of output values ​​less than the number of input values. A specific example is a maximum pooling process. The upsampling layer is a layer for performing processing by copying the set of input values ​​or adding values ​​interpolated from the set of input values ​​to make the number of output values ​​greater than the number of input values. Specific examples include linear interpolation processing. The merging layer is a layer for performing a process for inputting a plurality of sets of values ​​such as a set of output values ​​of a layer and a set of pixel values ​​constituting an image from a plurality of sources, and merging the plurality of sets of values ​​by connecting or adding the plurality of sets of values. With such a configuration, a set of values ​​output by passing a set of pixel values ​​constituting an input image 1301 through a convolution processing block is merged with a set of pixel values ​​constituting the input image 1301 in the merging layer. The merged set of pixel values ​​is then formed into a high image quality image 1302 in a final convolution layer. Although not shown in the figure, as a variation of the CNN configuration, a batch normalization layer and / or an activation layer using a rectified linear function (rectified linear unit) may be built into the convolution layer or otherwise incorporated after the convolution layer. In Figure 7 In the present invention, for the convenience of description, the processing target image is described as a two-dimensional image. However, the present invention is not limited thereto. The present invention also covers the case where a three-dimensional low image quality motion contrast image is input into an image quality enhancement unit to output a three-dimensional high image quality motion contrast image.

[0078] Now, a graphics processing unit (GPU) can efficiently perform calculations by processing more pieces of data in parallel. Therefore, using a GPU to perform processing is effective in training a learning model multiple times like deep learning. In this exemplary embodiment, the processing of an information processing device as an example of a learning unit (not shown) therefore uses a GPU in addition to a CPU. Specifically, in executing a training program including a learning model, the CPU and the GPU perform training by performing calculations in a collaborative manner. The calculations in the processing of the training unit can be performed by either the CPU or the GPU alone. Like the training unit, the image quality enhancement unit can also use a GPU. In addition, the training unit may include an error detection unit and an update unit not shown. The error detection unit obtains the error between the output data output from the output layer of the neural network in response to the input data input to the input layer and the true value data. The error detection unit can calculate the error between the output data from the neural network and the true value data by using a loss function. Based on the error obtained by the error detection unit, the update unit updates the coupling weight coefficients between the nodes of the neural network, etc. to reduce the error. The update unit updates the coupling weight coefficients, etc., for example, by back propagation. Back propagation is a technique for adjusting the coupling weight coefficients between the nodes of the neural network, etc., so that the aforementioned error is reduced.

[0079] The operator can give an instruction to start the OCTA analysis process by using the operation unit 104. In the present exemplary embodiment, double-clicking the Figure 6B The motion contrast image of the report screen is changed to the Fig. 6A The motion contrast image is displayed in enlarged form, and analysis processing can be performed using the information processing device. Any type of analysis processing can be performed. In the present exemplary embodiment, the analysis type indicated by density analysis 903 or by selecting Fig. 6A , and if necessary, select an item 912 related to the number of analysis dimensions to specify a desired analysis type. Thus, for example, based on an operator's instruction, the analysis process according to the aforementioned exemplary embodiment can be performed using a motion contrast image whose image quality is enhanced by a training model for image quality enhancement. For example, this can improve the accuracy of the analysis process according to the aforementioned exemplary embodiment.

[0080] Now, refer to Fig. 6A and Figure 6B The execution of the image quality enhancement process at the time of screen transition is described. Fig. 6A Shows an enlarged display Figure 6B An example of a report screen for an OCTA image. Fig. 6A In the example, button 911 is also similar to Figure 6BFor example, by double-clicking the OCTA image FIG. 6B to FIG. 6A By using the close button (not shown) to achieve the FIG. 6A to FIG. 6B Note that screen transition is not limited to the method described here, and a user interface not shown may be used.

[0081] If it is specified that the image quality enhancement process is executed when the image is switched (button 911 is activated), this state is maintained even after the screen is switched. Figure 6B The screen changes to Fig. 6A When the screen Figure 6B If the screen displays a high-quality image, Fig. 6A The screen of the display also displays a high image quality image. Then, button 911 remains activated. The same applies to FIG. 6A to FIG. 6B The transformation. Fig. 6A In the example, the display can be switched to a low image quality image by specifying button 911.

[0082] The screen transition is not limited to the screen transition described here, and as long as a transition to a screen displaying the same imaging data (such as a display screen for follow-up and a panoramic display screen) occurs, the state of displaying a high image quality image is maintained. In other words, an image corresponding to the state of the image quality enhancement button on the display screen before the transition is displayed on the display screen after the transition. For example, if the image quality enhancement button on the display screen before the transition is activated, a high image quality image is displayed on the display screen after the transition. For example, if the image quality enhancement button on the display screen before the transition is deactivated, a low image quality image is displayed on the display screen after the transition. If the image quality enhancement button on the display screen for follow-up (for example, the image quality enhancement button described below) is deactivated, the image quality enhancement button on the display screen for follow-up is deactivated. Figure 8 3420) is activated, a plurality of images acquired at different dates and times (on different examination dates) and arranged and displayed on the display screen for follow-up may be switched to high image quality images. In other words, the information processing device may be configured such that if the image quality enhancement button on the display screen for follow-up is activated, the activation is reflected in an integrated manner on a plurality of images acquired at different dates and times.

[0083] Figure 8 An example of a display screen for follow-up is shown. If tab 3801 is selected based on the inspector's instruction, Figure 8The display screen for follow-up is shown. Here, the examiner can change the depth range of the analysis target area by selecting from the predetermined depth range set displayed in the list boxes (3802 and 3803). For example, the retinal surface layer is selected in the list box 3802, and the retinal deep layer is selected in the list box 3803. The analysis results of the motion contrast image of the retinal surface layer are displayed in the upper display area. The analysis results of the retinal deep layer motion contrast image are displayed in the lower display area. In other words, if a depth range is selected, a plurality of images of different dates and times are switched integrally to a side-by-side display of the analysis results of a plurality of motion contrast images in the selected depth range.

[0084] Here, if the display of the analysis results is deselected, the display of the analysis results can be switched to a parallel display of a plurality of motion contrast images of different dates and times. Then, if button 3420 is designated based on the inspector's instruction, the display of the plurality of motion contrast images is switched to a display of a plurality of high image quality images. Fig. 6A and Figure 6B Like button 911 in , button 3420 is an example of an image quality enhancement button.

[0085] If the display of the analysis results is selected and button 3420 is specified based on the inspector's instruction, the display of the analysis results of the plurality of motion contrast images is switched to the display of the analysis results of the plurality of high image quality images as a whole. Here, the analysis results can be displayed by superimposing the analysis results on the image with a given transparency. Here, for example, the switching to the display of the analysis results can be achieved by switching to a state in which the analysis results are superimposed on the displayed image with a given transparency. For example, the switching to the display of the analysis results can be achieved by switching to the display of an image (e.g., a two-dimensional graph) obtained by performing a blending process on the analysis results and the image with a given transparency.

[0086] The layer boundary type and offset position for specifying the depth range can be changed integrally from a user interface such as list boxes 3805 and 3806. The depth range of a plurality of motion contrast images of different dates and times can be changed integrally by also displaying a tomographic image and moving the layer boundary data superimposed on the tomographic image based on the inspector's instruction. Here, a plurality of tomographic images of different dates and times can be displayed side by side, and if the aforementioned movement is performed on one tomographic image, the layer boundary data can be similarly moved on other tomographic images. In addition, for example, the image projection method and the presence or absence of projection artifact suppression processing can be changed by selecting from a user interface such as a context menu. The selection button 3807 can be selected to display a selection screen, and an image selected from the image list displayed on the selection screen can be displayed. Figure 8The arrow 3804 displayed at the top is a symbol indicating the currently selected examination, where the examination selected at the time of follow-up imaging ( Figure 8 The leftmost image in FIG. 1 is used as a reference check (baseline). It should be understood that a symbol indicating a reference check may be displayed on the display unit.

[0087] If the "Show Difference" check box 3808 is specified, the analysis value distribution (map, sector graph) of the baseline image is displayed on the baseline image. In this case, a differential analysis value graph between the analysis value distribution calculated for the baseline image and the analysis value distribution calculated for the image displayed in the area corresponding to the other inspection dates is displayed in each area. A trend graph (a graph of the analysis values ​​of images of each inspection date obtained by time change analysis) can be displayed on the report screen as an analysis result. In other words, time series data (e.g., a time series graph) about multiple analysis results corresponding to multiple images of different dates and times can be displayed. Here, analysis results of dates and times different from multiple dates and times corresponding to multiple display images can also be displayed as time series data in a manner that is distinguishable from multiple analysis results corresponding to multiple display images (e.g., points on the time series graph have different colors depending on the presence or absence of image display). The regression line (curve) and / or the corresponding equation of the trend graph can be displayed on the report screen.

[0088] The present exemplary embodiment has processed motion contrast images. However, this is not restrictive. The image to be subjected to processing (such as display, image quality enhancement, and image analysis) according to the present exemplary embodiment may be a tomographic image. The image is also not limited to the tomographic image, and other images such as SLO images, fundus photographs, and fluorescent fundus photographs may also be used. In this case, the user interface for performing image quality enhancement processing may include: a user interface for giving instructions for performing image quality enhancement processing on multiple different types of images, and a user interface for giving instructions for performing image quality enhancement processing on one or more images selected from multiple different types of images.

[0089] For example, the target image of the image quality enhancement process may be an OCTA frontal image corresponding to one depth range, rather than a plurality of OCTA frontal images (OCTA en-face image, motion contrast en-face image) corresponding to a plurality of depth ranges. The target image of the image quality enhancement process may be, for example, a brightness frontal image (brightness en-face image), a B-scan OCT tomographic image, or a B-scan tomographic image of motion contrast data (OCTA tomographic image), rather than an OCTA frontal image. The target image of the image quality enhancement process may include not only an OCTA frontal image, but also various medical images, including, for example, a brightness frontal image, a B-scan OCT tomographic image, and a B-scan tomographic image of motion contrast data (OCTA tomographic image). In other words, for example, the target image of the image quality enhancement process may be at least one of the various medical images displayed on the display screen of the display unit. Here, for example, since the feature amount of an image may differ depending on the type of image, a training model for image quality enhancement corresponding to each type of target image of the image quality enhancement process may be used. For example, the information processing device may be configured so that if button 911 or button 3420 is pressed, not only the image quality enhancement process is performed on the OCTA frontal image using the training model for image quality enhancement corresponding to the OCTA frontal image, but also the image quality enhancement process is performed on the OCT tomographic image using the training model for image quality enhancement corresponding to the OCT tomographic image. For example, the information processing device may be configured so that if button 911 or button 3420 is pressed, not only the display is switched to displaying a high-image-quality OCTA frontal image generated using the image quality enhancement training model corresponding to the OCTA frontal image, but also the display is switched to displaying a high-image-quality OCT tomographic image generated using the image quality enhancement training model corresponding to the OCT tomographic image. Here, the information processing device may be configured so that lines indicating the position of the OCT tomographic image are superimposed on the OCTA frontal image. These lines may be configured to be movable on the OCTA frontal image based on the inspector's instructions. The information processing apparatus may also be configured to switch to displaying a high-quality OCT tomographic image obtained by performing image quality enhancement processing on the OCT tomographic image corresponding to the current line position if display of button 911 or button 3420 is activated and the line is moved. The information processing apparatus may be configured to display an image quality enhancement button corresponding to button 3420 for each target image for image quality enhancement processing so that image quality enhancement processing can be performed independently on each image.

[0090] In addition, information indicating a blood vessel region in an OCTA tomographic image (e.g., motion contrast data higher than or equal to a threshold value) can be superimposed on an OCT tomographic image as a B-scan at a corresponding position. For example, if the OCT tomographic image here is enhanced in terms of image quality, the OCTA tomographic image at the corresponding position can be enhanced in terms of image quality. Then, information indicating a blood vessel region in an OCTA tomographic image acquired by image quality enhancement can be displayed in a superimposed manner on the OCT tomographic image acquired by image quality enhancement. The information indicating a blood vessel region can be any information recognizable in terms of color, etc. The information processing device can be configured so that the information indicating a blood vessel region can be switched between superimposed display and hiding based on an instruction of an examiner. If a line indicating the position of an OCT tomographic image is moved on an OCTA frontal image, the display of the OCT tomographic image can be updated based on the position of the line. Here, since the OCTA tomographic image at the corresponding position is also updated, the superimposed display of the information indicating a blood vessel region acquired from the OCTA tomographic image can be updated. For example, this makes it possible to effectively observe the three-dimensional distribution and state of a vascular region at a given position while easily confirming the positional relationship between the vascular region and the region of interest. The image quality of an OCTA tomographic image can be enhanced by performing image quality enhancement processing such as addition averaging processing on multiple OCTA tomographic images acquired at corresponding positions, instead of using a training model for image quality enhancement. The OCT tomographic image can be a pseudo-OCT tomographic image reconstructed as a cross section of the OCT volume data at a given position. The OCTA tomographic image can be a pseudo-OCTA tomographic image reconstructed as a cross section of the OCTA volume data at a given position. The given position can be at least one arbitrary position, and the information processing device can be configured so that the given position can be changed based on the inspector's instructions. Here, the information processing device can be configured to reconstruct multiple pseudo-tomographic images corresponding to multiple positions.

[0091] Only one or more tomographic images (for example, one or more OCT tomographic images, or one or more OCTA tomographic images) may be displayed. If a plurality of tomographic images are displayed, tomographic images acquired at various positions in the sub-scanning direction may be displayed. For example, if a plurality of tomographic images acquired by cross scanning or the like are displayed with enhanced image quality, images in various scanning directions may be displayed. For example, if a plurality of tomographic images acquired by radial scanning or the like are displayed with enhanced image quality, selected ones of the plurality of tomographic images (for example, two tomographic images at mutually symmetrical positions relative to a reference line) may be displayed. In addition, as Figure 8As shown, a plurality of tomographic images can be displayed on a display screen for follow-up, and an image quality enhancement instruction can be given and an analysis result (such as the thickness of a specific layer) can be displayed by using a technique similar to the aforementioned technique. Image quality enhancement processing can be performed on tomographic images based on information stored in a database using a technique similar to the aforementioned technique.

[0092] Similarly, when displaying an SLO fundus image with enhanced image quality, for example, an SLO fundus image displayed on the same display screen can be displayed with enhanced image quality. For example, when displaying a luminance frontal image with enhanced image quality, a luminance frontal image displayed on the same display screen can be displayed with enhanced image quality. In addition, it is possible to Figure 8 A plurality of SLO fundus images or brightness front images are displayed on a display screen for follow-up such as shown, and an image quality enhancement instruction can be given by using a technique similar to the aforementioned technique and an analysis result (such as the thickness of a specific layer) can be displayed. Image quality enhancement processing can be performed on SLO fundus images and brightness front images based on information stored in a database using a technique similar to the aforementioned technique. Note that the display of tomographic images, SLO fundus images, and brightness front images is illustrative, and such images can be displayed in any mode according to the desired configuration. At least two or more of the OCTA front image, tomographic image, SLO fundus image, and brightness front image can be enhanced in image quality and displayed based on a single instruction.

[0093] By such a configuration, the display control unit 103 can display the image processed by the image quality enhancement unit (not shown) according to the present exemplary embodiment on the display unit. As described above, if at least one of a plurality of conditions related to the display of a high image quality image, the display of an analysis result, the depth range of a front image to be displayed, etc. is selected, the selected state can be maintained even after the display screen is switched.

[0094] As described above, if at least one of the multiple conditions is selected, the state of selecting the at least one condition can be maintained even after another condition is selected. For example, if the display of the analysis result is selected, the display control unit 103 can switch the display of the analysis result of the low image quality image to the display of the analysis result of the high image quality image based on the inspector's instruction (for example, when the button 911 or the button 3420 is specified). If the display of the analysis result is selected, the display control unit 103 can switch the display of the analysis result of the high image quality image to the display of the analysis result of the low image quality image based on the inspector's instruction (for example, when the button 911 or the button 2420 is not specified).

[0095] If the display of the high image quality image is deselected, the display control unit 103 may switch the display of the analysis result of the low image quality image to the display of the low image quality image based on the instruction of the inspector (for example, when the display of the analysis result is not specified). If the display of the high image quality image is deselected, the display control unit 103 may switch the display of the low image quality image to the display of the analysis result of the low image quality image based on the instruction of the inspector (for example, when the display of the analysis result is specified). If the display of the high image quality image is selected, the display control unit 103 may switch the display of the analysis result of the high image quality image to the display of the high image quality image based on the instruction of the inspector (for example, when the display of the analysis result is not specified). If the display of the high image quality image is selected, the display control unit 103 may switch the display of the high image quality image to the display of the analysis result of the high image quality image based on the instruction of the inspector (for example, when the display of the analysis result is specified).

[0096] Assume a case where the display of high-image quality images is deselected and the display of the first type of analysis results is selected. In this case, the display control unit 103 can switch the display of the first type of analysis results of low-image quality images to the display of the second type of analysis results of low-image quality images based on the inspector's instructions (for example, when the display of the second type of analysis results is specified). Now, assume a case where the display of high-image quality images is selected and the display of the first type of analysis results is selected. In this case, the display control unit 103 can switch the display of the first type of analysis results of high-image quality images to the display of the second type of analysis results of high-image quality images based on the inspector's instructions (for example, when the display of the second type of analysis results is specified).

[0097] The display screen for follow-up can be configured so that such changes in the display are reflected in a unified manner on multiple images obtained at different dates and times as described above. Here, the analysis result can be displayed by superimposing it on the image with a given transparency. For example, switching to the display of the analysis result can be achieved by switching to a state in which the analysis result is superimposed on the displayed image with a given transparency. For example, switching to the display of the analysis result can be achieved by switching to the display of an image (e.g., a two-dimensional graph) obtained by mixing the analysis result and the image with a given transparency.

[0098] In the aforementioned exemplary embodiment, the display control unit 103 can display an image selected from the high image quality image generated by the image quality enhancement unit and the input image based on the inspector's instruction on the display unit. The display control unit 103 can also switch the display on the display screen of the display unit from the captured image (input image) to the high image quality image based on the inspector's instruction. In other words, the display control unit 103 can switch the display of the low image quality image to the display of the high image quality image based on the inspector's instruction. The display control unit 103 can also switch the display of the high image quality image to the display of the low image quality image based on the inspector's instruction.

[0099] In addition, the image quality enhancement unit of the information processing device can start the image quality enhancement process using the image quality enhancement engine (training model for image quality enhancement) (the image can be input to the image quality enhancement engine) based on the inspector's instructions, and the display control unit 103 can display the high image quality image generated by the image quality enhancement unit on the display unit. Alternatively, when the input image is captured by the imaging device (OCT), the image quality enhancement engine can automatically generate a high image quality image based on the input image, and the display control unit 103 can display the high image quality image on the display unit based on the inspector's instructions. Here, the image quality enhancement engine includes a training model that performs the aforementioned image quality improvement process (image quality enhancement process).

[0100] Such processing can also be performed similarly on the output of the analysis results. More specifically, the display control unit 103 can switch the display of the analysis results of the low image quality image to the display of the analysis results of the high image quality image based on the instructions of the inspector. The display control unit 103 can switch the display of the analysis results of the high image quality image to the display of the analysis results of the low image quality image based on the instructions of the inspector. It should be understood that the display control unit 103 can switch the display of the analysis results of the low image quality image to the display of the low image quality image based on the instructions of the inspector. The display control unit 103 can switch the display of the low image quality image to the display of the analysis results of the low image quality image based on the instructions of the inspector. The display control unit 103 can switch the display of the analysis results of the high image quality image to the display of the high image quality image based on the instructions of the inspector. The display control unit 103 can switch the display of the high image quality image to the display of the analysis results of the high image quality image based on the instructions of the inspector.

[0101] The display control unit 103 can also switch the display of the analysis result of the low image quality image to the display of another type of analysis result of the low image quality image based on the inspector's instruction. The display control unit 103 can also switch the display of the analysis result of the high image quality image to the display of another type of analysis result of the high image quality image based on the inspector's instruction.

[0102] Here, the analysis result of the high image quality image may be displayed by superimposing it on the high image quality image with a given transparency. The analysis result of the low image quality image may be displayed by superimposing it on the low image quality image with a given transparency. For example, switching to the display of the analysis result may be achieved by switching to a state in which the analysis result is superimposed on the displayed image with a given transparency. For example, switching to the display of the analysis result may be achieved by switching to the display of an image (e.g., a two-dimensional graph) obtained by mixing the analysis result and the image with a given transparency.

[0103] (First Modification)

[0104] The information processing apparatus according to the aforementioned exemplary embodiment has an automatic transmission function of automatically transmitting imaging data captured by an ophthalmic imaging device or data converted into a specified format to a pre-specified storage location. Possible use cases of the automatic transmission function in actual hospital operations include cases where data is transmitted to a recording and archiving system (the so-called picture archiving and communication system (PACS)) and cases where data is transmitted to electronic medical records and other diagnostic systems. In such operations, it is sometimes necessary to transmit different data contents depending on the ophthalmic imaging device. For example, when transmitting imaging data to a recording and archiving system, fundus images are often transmitted from a fundus camera, and tomographic images are transmitted from an OCT. When transmitting data to a diagnostic system, fundus images are often transmitted from a fundus camera, and report images including analysis results such as retinal thickness maps are transmitted from the OCT, rather than tomographic images. For example, the information processing apparatus according to the aforementioned exemplary embodiment can even handle such use cases.

[0105] In the aforementioned exemplary embodiment, the information processing device may be configured so that the examiner can manually transmit (manually transfer) a single piece of imaging data from a display screen such as a report screen. For example, the information processing device may be configured so that if a manual transmission button is pressed on the report screen based on the examiner's instructions, the image displayed on the report screen or the report image corresponding to the report screen is transmitted as imaging data. For example, if an examination is specified on the patient screen based on the examiner's instructions, the imaging data related to the specified examination can be made available for transmission based on the examiner's instructions. If a patient is specified on the patient screen based on the examiner's instructions, the imaging data related to at least one examination of the specified patient can be made available for transmission based on the examiner's instructions.

[0106] The information processing device may be configured such that: if a button (image quality enhancement button) for giving an instruction to display a high image quality image is set to be activated (image quality enhancement processing is enabled) by default on the initial display screen of the report screen, a report image corresponding to the report screen including the high image quality image, etc. is transmitted to the server based on the instruction of the examiner. The information processing device may be configured such that: if a button for giving an instruction to display a high image quality image is set to be activated by default, at the end of the examination (for example, when the imaging confirmation screen or the preview screen is switched to the report screen based on the instruction of the examiner), the report image corresponding to the report screen including the high image quality image, etc. is (automatically) transmitted to the server. Here, the information processing device may be configured such that a report image generated based on various settings included in the default settings (for example, settings related to at least one of the following: a depth range for generating an en-face image on the initial display screen of the report screen, the presence or absence of superposition of an analysis map, whether the image is a high image quality image, and whether the display screen is for follow-up) is transmitted to the server.

[0107] (Second Modification)

[0108] In the aforementioned exemplary embodiment, the types of transmittable images in the transmission setting may include not only at least one tomographic imaging image (B-scan image) but also one or more frontal images ((one or more) en-face images). Here, data including a high image quality image (a second medical image) and a low image quality image obtained from a low image quality image (a first medical image) corresponding to the imaging data by using a training model for image quality enhancement (an image quality enhancement model, an image quality enhancement engine) may be transmittable as a set. Here, the data may be transmittable to a training data server for additional training. Managing the data in the form of the aforementioned set facilitates use of the set as training data when generating a training model for image quality enhancement even on a server that is not intended for such a purpose. The training model for image quality enhancement may be a training model (a machine learning model, a machine learning engine) obtained by performing machine learning using training data including a low image quality image as input data and a high image quality image as true value data (teaching data).

[0109] The aforementioned training model can be obtained by machine learning using training data. Examples of machine learning include deep learning using a multi-level neural network. For example, a convolutional neural network (CNN) can be used as a machine learning model for at least a portion of a multi-level neural network. Techniques associated with autoencoders can be used for at least a portion of a multi-level neural network. Techniques associated with back propagation can be used for training. Note that machine learning is not limited to deep learning, and any learning using the following model can be adopted, which can extract (express) the feature quantity of training data such as images by itself through training. A training model refers to a machine learning model that is pre-trained with appropriate training data based on a given machine learning algorithm. Note that a training model should refer to a model that can be additionally trained, rather than a model that is no longer further trained. The training data includes paired input data and output data (true value data). As used herein, the training data can be referred to as teaching data, and the true value data can be referred to as teaching data. The image quality enhancement engine can be a training model obtained by additional training using training data including at least one high image quality image generated by the image quality enhancement engine. Here, the information processing apparatus may be configured such that whether to use the high image quality image as training data for additional training can be selected based on an inspector's instruction.

[0110] (Third Modification)

[0111] The display control unit 103 in the aforementioned various exemplary embodiments and modified examples can display analysis results such as the thickness of the desired layer and various blood vessel densities on a report screen as a display screen. As an analysis result, the display control unit 103 can also display parameter values ​​(distributions) related to a region of interest including at least one of the following items: optic disc, macula, blood vessel area, nerve fiber bundle, vitreous body area, macula area, choroid area, sclera area, lamina cribrosa area, retinal layer boundary, retinal layer boundary end, visual cell, blood cell, blood vessel wall, blood vessel inner wall boundary, blood vessel outer wall boundary, ganglion cell, corneal area, corner area, and Schlemm's canal. Here, for example, accurate analysis results can be displayed by analyzing medical images to which various types of artifact reduction processing are applied. Examples of artifacts may include false image areas that appear due to light absorption in blood vessel areas, projection artifacts, and band-shaped artifacts that appear in the main scanning direction of the measurement light in the frontal image due to the state of the eye to be examined (such as movement or blinking). Artifacts may refer to any imaging error area that randomly appears in a medical image of a predetermined area of ​​the subject in each imaging operation. It is also possible to display parameter values ​​(distributions) associated with an area including at least one of the aforementioned various artifacts (imaging error areas) as analysis results. It is also possible to display parameter values ​​(distributions) associated with an area including at least one of abnormal areas such as vitreous, neovascularization, leukoplakia (hard leukoplakia), and pseudodrusen as analysis results. It is possible to display comparison results obtained by comparing a standard value or a standard range obtained using a standard database with the analysis results.

[0112] The analysis result may be displayed as an analysis map, a sector indicating a corresponding statistical value in each segment, etc. The analysis result may be an analysis result generated by using a training model (analysis result generation engine, training model for analysis result generation) obtained by training the analysis result of a medical image as training data. Here, the training model may be a model obtained by training using training data including a medical image and the analysis result of the medical image, training data including a medical image and the analysis result of a medical image of a different type from the medical image, etc.

[0113] The training model may be a model obtained by training using training data including input data having a plurality of different types of medical images of a predetermined part as a set, such as a brightness en-face image (brightness tomographic image) and a motion contrast en-face image. Here, the brightness en-face image corresponds to the tomographic en-face image, and the motion contrast en-face image corresponds to the OCTA en-face image.

[0114] The information processing device may be configured to display an analysis result obtained by using a high image quality image generated by an image quality enhancement engine. The training model for image quality enhancement may be a model obtained by training using training data including a first image as input data and a second image having a higher image quality than the first image as true value data. For example, the second image may be a high image quality image in which contrast is enhanced and noise is reduced by superposition processing of multiple first images (e.g., averaging processing of multiple aligned first images) or the like.

[0115] The input data included in the training data may include a high image quality image or a set of low and high image quality images generated by the image quality enhancement engine.

[0116] For example, the training data may be data including labeled (annotated) input data having information including at least one of the following as true value data (for supervised learning): an analysis value (such as an average and a median) obtained by analyzing an analysis area, a table containing analysis values, an analysis map, and the location of the analysis area such as a sector in an image. The information processing apparatus may be configured to display the analysis result obtained by the training model for analysis result generation based on an inspector's instruction.

[0117] The display control unit 103 in the aforementioned exemplary embodiments and modified examples can display various diagnostic results, such as diagnostic results of glaucoma and age-related macular degeneration, on a report screen as a display screen. Here, for example, by analyzing a medical image to which the aforementioned various types of artifact reduction processing are applied, an accurate diagnostic result can be displayed. The position of the identified abnormal area can be displayed on the image as a diagnostic result. Characters, etc. can be used to display the state of the abnormal area, etc. The classification results of the abnormal area (such as Curtin classification) can be displayed as a diagnostic result. Information indicating the possibility of each abnormal area (for example, a numerical value indicating a ratio) can be displayed as a classification result. Information required for a doctor to make a diagnosis can be displayed as a diagnostic result. Examples of the required information include recommendations for additional imaging, etc. For example, if an abnormal area is detected in a blood vessel area in an OCTA image, a message for additional fluorescent imaging using a contrast agent that can observe blood vessels in more detail than OCTA can be displayed. The diagnostic result may be information about the subject's future diagnostic plan. The diagnosis result may be information including, for example, at least one of the following items: a diagnosis name, a type and state (extent) of a lesion (abnormal area), a position of the lesion in an image, a position of the lesion relative to an area of ​​interest, findings (radiograph interpretation findings, etc.), a basis for the diagnosis name (such as affirmative medical support information), and a basis for denying the diagnosis name (negative medical support information). Here, for example, a diagnosis result that is more likely than a diagnosis name input based on an inspector's instruction may be displayed as medical support information. If multiple types of medical images are used, one or more types of medical images that can serve as a basis for the diagnosis result may be displayed in a recognizable manner.

[0118] The diagnosis result may be a diagnosis result generated by using a training model (diagnosis result generation engine, training model for diagnosis result generation) trained using the diagnosis result of the medical image as training data. The training model may be a training model obtained by training using training data including medical images and the diagnosis result of the medical images, training data including medical images and the diagnosis result of medical images of a different type from the medical images, and the like. The information processing apparatus may be configured to display the diagnosis result obtained by using a high image quality image generated by the image quality enhancement engine.

[0119] The input data included in the training data may be a high image quality image generated by the image quality enhancement engine, or a collection of low and high image quality images. The training data may be data including labeled (annotated) input data having information including at least one of the following items as true value data (for supervised learning): diagnosis name, type and state (extent) of lesion (abnormal area), location of lesion in the image, location of lesion relative to the region of interest, findings (radiograph interpretation findings, etc.), basis for diagnosis name (such as affirmative medical support information), and basis for negative diagnosis name (negative medical support information). The information processing device may be configured to display the diagnosis result obtained by the training model for diagnosis result generation based on the inspector's instructions.

[0120] The aforementioned various training models can be trained not only by supervised learning (training with labeled training data) but also by semi-supervised learning. Semi-supervised learning is a technique that includes, for example, training each of a plurality of discriminators (classifiers) by supervised learning, then discriminating (classifying) unlabeled training data, automatically marking (annotating) the discrimination results (classification results) based on their reliability (for example, discriminating results with a labeling probability higher than or equal to a threshold), and performing training with labeled training data. An example of semi-supervised learning may be collaborative training (multi-view). For example, a training model for diagnostic result generation may be a training model obtained by performing semi-supervised learning (for example, collaborative training) using a first discriminator for discriminating medical images of normal test subjects and a second discriminator for discriminating medical images including specific lesions. For example, the training model is not limited to diagnostic purposes and may be dedicated to imaging assistance, etc. In this case, the second discriminator may be a discriminator for discriminating medical images including partial areas such as regions of interest and artifact regions.

[0121] The display control unit 103 in the aforementioned various exemplary embodiments and modified examples can display the object recognition result (object detection result) or the segmentation result of a partial area (such as a region of interest, an artifact region, and the above-mentioned abnormal region) on a report screen as a display screen. Here, for example, a rectangular frame or the like can be displayed in an overlay manner around an object on the image. For example, a color or the like can be displayed in an overlay manner on an object in the image. The object recognition result or the segmentation result can be a result generated using a training model (object recognition engine, a training model for object recognition, a segmentation engine, or a training model for segmentation), which is obtained by training using training data, and the training data includes a medical image marked (annotated) with information indicating object recognition or segmentation as true value data. The aforementioned analysis result or diagnosis result can be obtained by using the aforementioned object recognition result or segmentation result. For example, a process for generating an analysis result or a diagnosis result can be performed on a region of interest obtained by object recognition or segmentation processing.

[0122] In order to detect abnormal areas, the information processing device may use a generative adversarial network (GAN) or a variational autoencoder (VAE). For example, a deep convolutional GAN ​​(DCGAN) including a generator trained to generate a tomographic image and a discriminator trained to identify a new tomographic image generated by the generator from an actual tomographic image may be used as a machine learning model.

[0123] For example, in the case of using DCGAN, the discriminator encodes the input tomographic image as a latent variable, and the generator generates a new tomographic image based on the latent variable. Then, the difference between the input tomographic image and the generated new tomographic image can be extracted (detected) as an abnormal area. For example, in the case of using VAE, the encoder encodes the input tomographic image as a latent variable, and the decoder decodes the latent variable to generate a new tomographic image. Then, the difference between the input tomographic image and the generated new tomographic image can be extracted as an abnormal area. Although the tomographic image is described as an example of input data, fundus images, anterior frontal images, etc. can also be used.

[0124] The information processing device can also detect abnormal areas using a convolutional autoencoder (CAE). In the case of using CAE, the CAE is trained using the same image as the input data and the output data. Therefore, if an image including an abnormal area is input to the CAE for estimation, an image excluding the abnormal area is output based on the training trend. Then, the difference between the image input to the CAE and the image output from the CAE can be extracted as the abnormal area. Even in this case, as input data, not only tomographic images but also fundus images, anterior frontal images, etc. can be used.

[0125] In this case, the information processing device can generate information about the difference between the medical image obtained by using a generative adversarial network or an autoencoder and the medical image input to the generative adversarial network or the autoencoder as information about the abnormal area. Therefore, it can be expected that the information processing device can detect the abnormal area at high speed and accurately. For example, even if it is difficult to collect a large number of medical images including abnormal areas as training data to improve the detection accuracy of abnormal areas, medical images of normal test subjects that are relatively easy to collect in quantity can be used as training data. This achieves, for example, effective training for detecting abnormal areas with high accuracy. As adopted herein, the autoencoder includes VAE and CAE. At least a part of the generation unit in the generative adversarial network can be composed of VAE. This achieves, for example, the generation of relatively clear images while reducing the phenomenon of generating similar data. For example, the information processing device can generate information about the difference between medical images generated from various medical images by using a generative adversarial network or an autoencoder and medical images input to the generative adversarial network or the autoencoder as information about the abnormal area. For example, the display control unit 103 can display information about the difference between a medical image generated from various medical images by using a generative adversarial network or an autoencoder and a medical image input to the generative adversarial network or the autoencoder on the display unit as information about the abnormal area.

[0126] Depending on the type of disease, the diseased eye has different image features. Therefore, the training model used in the aforementioned various exemplary embodiments and modifications can be generated and provided for each type of disease or each abnormal area. In this case, for example, the information processing device can select one or more training models to be used for processing based on the operator's input (instruction) about the disease type, abnormal area, etc. of the eye to be examined. Note that the training model prepared for each type of disease or each abnormal area is not limited to a model for detecting the retinal layer and generating an image of a region label, etc. For example, a training model used as an image evaluation engine, an analysis engine, etc. can be prepared. Here, the information processing device can identify the disease type and abnormal area of ​​the eye to be examined from the image by using a separately prepared training model. In this case, the information processing device can automatically select a training model to be used for the aforementioned processing based on the disease type and abnormal area identified using a separately prepared training model. The training model for identifying the disease type and abnormal area of ​​the eye to be examined can be trained using paired training data, with a tomographic image, a fundus image, etc. as input data and the disease type and abnormal area in such an image as output data. The tomographic image, the fundus image, etc. can be used as input data in the training data. A combination of these images can be used as input data.

[0127] Specifically, the training model used for generating the diagnosis result may be a training model obtained by training using training data, the training data including a plurality of different types of medical images of a predetermined area of ​​the subject as a set of input data. Examples of the input data included in the training data may include a motion contrast front image and a brightness front image (or brightness tomographic image) of the fundus as a set of input data, and a tomographic image (B-scan image) and a color fundus image (or fluorescent fundus image) of the fundus as a set of input data. Any plurality of different types of medical images obtained by different modes, different optical systems, different principles, etc. may be used.

[0128] Specifically, the training model used for generating the diagnosis result may be a training model obtained by training using training data, the training data including a plurality of medical images of different regions of the subject as a set of input data. Examples of the input data included in the training data may include a tomographic image (B-scan image) of the fundus and a tomographic image (B-scan image) of the anterior eye as a set of input data, and a three-dimensional OCT image of the fundus macula and a circular scan (or raster scan) tomographic image of the optic disc in the fundus as a set of input data.

[0129] The input data included in the training data may be a plurality of different types of medical images of different regions of the subject. An example of the input data included in the training data here may be input data including a tomographic imaging image of the anterior eye portion and a color fundus image as a set. In addition, the aforementioned various training models may be models obtained by training using training data, the training data including a plurality of medical images of a predetermined region of the subject at different imaging perspectives as a set of input data. The input data included in the training data may be a plurality of connected medical images obtained by dividing a predetermined region into a plurality of regions in a time-division manner like a panoramic image. Here, since using a wide-angle image like a panoramic image as training data provides a greater amount of information than a narrow-angle image, an accurate image feature quantity may be obtained. Therefore, the results of each process may be improved. The input data included in the training data may be input data including a plurality of medical images of a predetermined region of the subject at different dates and times as a set of input data.

[0130] The display screen displaying at least one of the aforementioned analysis results, diagnosis results, object recognition results, and segmentation results is not limited to the report screen. For example, such a display screen may be displayed as at least one of the following items: an imaging confirmation screen, a display screen for follow-up, and a preview screen for various adjustments before imaging (a display screen displaying various types of live motion images). For example, at least one of the results obtained by using the aforementioned various training models may be displayed on the imaging confirmation screen, so that the inspector may observe accurate results even immediately after imaging. For example, the information processing device may be configured to: if a specific object is identified, a frame surrounding the identified object is superimposed on the live motion image. Here, if the information indicating the possibility of the object recognition result (e.g., a numerical value indicating a ratio) exceeds a threshold value, the frame surrounding the object may be highlighted, for example, in a different color, etc. Therefore, the inspector may easily identify the object on the live motion image. The display switching between the aforementioned low image quality image and the high image quality image may be a switching between the analysis results of the low image quality image and the analysis results of the high image quality image.

[0131] The aforementioned various training models can be obtained by machine learning using training data. An example of machine learning is deep learning using a multi-level neural network. For example, a convolutional neural network (CNN) can be used as a machine learning model in at least a portion of a multi-layer neural network. Techniques related to autoencoders can be used for at least a portion of a multi-layer neural network. Techniques related to back propagation can be used for training. Techniques (dropout) for randomly deactivating units (neurons or nodes) can be used for training. Techniques (batch normalization) for normalizing the data delivered to each layer of a multi-layer neural network before applying an activation function (e.g., a rectified linear unit (ReLU) function) can be used for training. Note that machine learning is not limited to deep learning, and any learning that utilizes a model that can extract (represent) the feature quantity of training data such as images by itself through training can be used. As used herein, a machine learning model refers to a learning model using a machine learning algorithm such as deep learning. A training model refers to a machine learning model pre-trained using a given machine learning algorithm, for example, appropriate training data. Note that a training model refers to a model that is capable of additional training, rather than a model that is no longer further trained. The training data includes paired input data and output data (true value data). As employed herein, training data may be referred to as teaching data, and ground truth data may be referred to as training data.

[0132] GPU can efficiently perform calculations by processing more data in parallel. Therefore, like deep learning, using GPU to perform processing is effective in training learning models multiple times. In this variant, the processing of the information processing device as an example of a training unit (not shown) uses GPU in addition to CPU. Specifically, when executing a training program including a learning model, the CPU and GPU perform training by performing calculations in a collaborative manner. The calculations in the processing of the training unit can be performed by the CPU or GPU alone. Like the training unit, the processing unit (estimation unit) that performs processing using the aforementioned various training models can also use GPU. In addition, the training unit may include an error detection unit and an update unit not shown. The error detection unit obtains the error between the output data output from the output layer of the neural network in response to the input data input to the input layer and the true value data. The error detection unit can calculate the error between the output data from the neural network and the true value data by using a loss function. Based on the error obtained by the error detection unit, the update unit updates the coupling weight coefficients between the nodes of the neural network to reduce the error. The update unit updates the coupling weight coefficients, etc., for example, by back propagation. Back propagation is a technique for adjusting the coupling weight coefficients, etc. between each node of the neural network so that the aforementioned error is reduced.

[0133] A U-net machine learning model having an encoder function having a multi-level layer including a plurality of downsampling layers and a decoder function having a multi-level layer including a plurality of upsampling layers can be applied to a machine learning model for image quality enhancement, segmentation, etc. The U-net machine learning model is configured so that position information (spatial information) blurred in the multi-level layer constituting the encoder can be used at the same order level (mutually corresponding levels) in the multi-level layer constituting the decoder (for example, by using skip connections).

[0134] For example, a fully convolutional network (FCN), SegNet, etc. can be used as a machine learning model for image quality enhancement, segmentation, etc. A machine learning model for performing object recognition in units of regions based on a desired configuration can be used. For example, a regional CNN (RCNN), a fast RCNN, or a faster RCNN can be used as a machine learning model for performing object recognition. In addition, you can also use only look once (YOLO) or a single-stage detector or a single-stage multi-box detector (SSD) as a machine learning model for performing object recognition in units of regions.

[0135] Examples of machine learning models may include capsule networks (CapsNet). In a typical neural network, for example, each unit (each neuron or each node) is configured to output a scalar value so that the spatial information about the spatial position relationship (relative position) between image features is reduced. This enables, for example, training to reduce the effects of local distortion, translation, etc. in the image. In contrast, in a capsule network, for example, each unit (each capsule) is configured to output spatial information as a vector so that the spatial information is maintained. For example, this enables training to be performed while taking into account the spatial position relationship between image features.

[0136] The image quality enhancement engine (training model for image quality enhancement) may be a training model obtained by performing additional training using training data including at least one high image quality image generated by the image quality enhancement engine. Here, the information processing device may be configured to select whether to use a high image quality image as training data for additional training based on the instructions of the inspector. Such a configuration is not limited to the training model for image quality enhancement, and may be applied to the aforementioned various training models. The true value data used in training the aforementioned various training models may be generated by using a training model for true value data generation (for generating true value data such as labeled (annotated) data). Here, the training model for true value data generation may be a model obtained by performing (sequential) additional training using true value data labeled (annotated) by the inspector. More specifically, the training model for true value data generation may be a model obtained by performing additional training using training data with unlabeled data as input data and labeled data as output data. The information processing device may be configured to correct the results of object recognition, segmentation, etc. in a frame determined to have low result accuracy in a plurality of consecutive frames such as frames of a moving image, taking into account the results in the previous frame and the subsequent frame. Here, the information processing apparatus may be configured to perform additional training based on the inspector's instruction, with the correction result being used as true value data.

[0137] In the aforementioned various exemplary embodiments and variations, when detecting partial areas of the eye to be inspected (such as areas of interest, artifact areas, and abnormal areas) by using a training model for object recognition or a training model for segmentation, predetermined image processing can be applied to each detection area. For example, assume a case where at least two partial areas including a vitreous area, a retinal area, and a choroidal area are detected. In this case, when imaging processing such as contrast adjustment is applied to at least two detection areas, adjustments suitable for each area can be made by using different image processing parameters. Displaying images with appropriate adjustments to each area enables the operator to make a more appropriate diagnosis of a disease, etc., area by area. Note that the configuration of using different image processing parameters for each detection area can also be similarly applied to areas of the eye to be inspected that are detected without using a training model.

[0138] (Fourth Modification)

[0139] The information processing device may be configured so that the aforementioned various training models are used for at least each frame of the live motion image in the preview screen in the aforementioned various exemplary embodiments and variations. Here, the information processing device may be configured so that if the preview screen displays multiple live motion images of different regions or multiple different types of live motion images, each corresponding training model is used for the live motion image. For example, this can even reduce the processing time of the live motion image, and the examiner can obtain accurate information before starting imaging. For example, since re-imaging failures can be reduced, etc., the diagnostic accuracy and efficiency can be improved.

[0140] Examples of multiple live motion images may include motion images of the anterior eye portion aligned in the X, Y, and Z directions and frontal motion images of the fundus for focus adjustment of the fundus observation optical system or for OCT focus adjustment. For example, multiple live motion images may be fundus tomographic motion images for OCT coherence gate adjustment (to adjust the difference in optical path length between the measured optical path length and the reference optical path length), etc. Here, the information processing device may be configured to perform the aforementioned various adjustments so that the area detected using the aforementioned training model for object recognition or the training model for segmentation meets the predetermined condition. For example, the information processing device may be configured to perform various adjustments, such as OCT focus adjustment, so that the values ​​(such as contrast values ​​and intensity values) related to the vitreous region or the predetermined retinal layer such as the retinal pigment epithelium (RPE) detected using the training model for object recognition or the training model for segmentation exceed a threshold value (or peak value). For example, the information processing device may be configured to perform OCT coherence gate adjustment so that the vitreous region or the predetermined retinal layer such as the RPE detected using the training model for object recognition or the training model for segmentation reaches a predetermined position in the depth direction.

[0141] In these cases, the image quality enhancement unit (not shown) of the information processing device can generate a high-image quality motion image by performing image quality enhancement processing on the motion image using a training model. In addition, the control unit (not shown) of the information processing device can control the drive of an optical component (such as an OCT reference mirror (not shown)) for changing the imaging range so that one of the different areas identified by segmentation processing, etc. reaches a predetermined position in the display area where the high-image quality motion image is displayed. In this case, the control unit can automatically perform alignment processing based on high-precision information so that the desired area reaches a predetermined position in the display area. For example, the optical component for changing the imaging range can be an optical component for adjusting the coherence gate position, and more specifically, a reference mirror, etc. The coherence gate position can be adjusted by changing the optical path length difference between the measurement optical path length and the reference optical path length. Examples of such optical components may include a mirror not shown for changing the optical path length of the measurement light. For example, the optical component for changing the imaging range may be a stage unit (not shown) of the imaging device. The control unit may control the driving of the scanning unit for scanning the measuring light so that, based on the instruction related to the start of imaging, a partial area (such as an artifact area) obtained by the segmentation processing, etc. is captured again (re-scanned) during the imaging or at the end of the imaging. In addition, for example, if the information indicating the possibility of the object recognition result related to the region of interest (for example, a numerical value indicating the ratio) exceeds a threshold value, the information processing device may be configured to automatically make various adjustments and start imaging, etc. The information processing device may be configured, for example, to switch to a state in which various adjustments can be made and imaging can be started based on the inspector's instruction if the information indicating the possibility of the object recognition result related to the region of interest (for example, a numerical value indicating the ratio) exceeds a threshold value.

[0142] For example, the motion images to which the aforementioned various training models can be applied are not limited to live motion images, and may be motion images stored (saved) in a storage unit. Here, for example, a motion image obtained by performing alignment at least for each frame of a fundus tomography motion image stored (saved) in a storage unit can be displayed on a display screen. For example, in order to observe the vitreous body in a suitable manner, a reference frame can be first selected with reference to the condition that the frame covers the vitreous body as much as possible. Here, each frame is a tomographic image (B-scan image) in the X and Z directions. Then, a motion image obtained by aligning other frames with the selected reference frame in the X and Z directions can be displayed on a display screen. Here, for example, the information processing device can be configured to continuously display a high image quality image (high image quality frame) sequentially generated by an image quality enhancement engine from at least each frame of the motion image.

[0143] As a technique for the aforementioned frame-to-frame alignment, the same technique may be applied to the alignment in the X direction and the alignment in the Z direction (depth direction), or different techniques may be applied. Alignment in the same direction may be performed multiple times by using different techniques. For example, precise alignment may be performed after rough alignment. Examples of alignment techniques include (rough Z direction) alignment using the retinal layer boundary obtained by segmentation processing of the tomographic image (B-scan image), (precise X direction and Z direction) alignment using correlation information (similarity) between multiple regions obtained by segmenting the tomographic image and the reference image, (X direction) alignment using a one-dimensional projection image generated for each tomographic image (B-scan image), and (X direction) alignment using a two-dimensional frontal image. The information processing device may be configured to perform rough alignment in units of pixels and then perform precise alignment in units of sub-pixels.

[0144] During various adjustments, imaging targets such as the retina of the eye to be examined may not have been successfully imaged. Therefore, due to the large difference between the medical images input to the training model and the medical images used as training data, accurate high image quality images cannot be obtained. Therefore, if the image quality evaluation or other evaluation value of the tomographic imaging image (B-scan) exceeds a threshold value, the information processing device can be configured to automatically start displaying a high image quality motion image (continuously displaying high image quality frames). The information processing device can be configured to make the image quality enhancement button specifiable (activated) by the examiner if the image quality evaluation or other evaluation value of the tomographic imaging image (B-scan) exceeds a threshold value. The image quality enhancement button is a button for specifying the execution of an image quality enhancement process. It should be understood that the image quality enhancement button can be a button for giving an instruction to display a high image quality image.

[0145] The information processing apparatus may be configured to prepare different training models for image quality enhancement for respective imaging modes having different scanning patterns, etc., and select a training model for image quality enhancement corresponding to the selected imaging mode. A single training model for image quality enhancement obtained by training using training data including various medical images obtained in different imaging modes may be used.

[0146] (Fifth Modification)

[0147] In the aforementioned various exemplary embodiments and variations, if the training model is undergoing additional training, it may be difficult to output (estimate, predict) by using the training model itself that is being additionally trained. Therefore, the information processing device is preferably configured to prohibit the input of medical images other than training data into the training model under additional training. In addition, another training model that is the same as the training model before performing additional training can be prepared as a backup training model. Here, the information processing device can be configured so that medical images other than training data can be input into the backup training model while additional training is in progress. After the additional training is completed, the training model of the additional training can be evaluated, and if no problems are found, the backup training model can be replaced by the training model of the additional training. If any problems are found, the backup training model can be used. For example, the training model of the additional training can be evaluated by using a training model for classification, which is used to classify high image quality images obtained by the training model for image quality enhancement from other types of images. For example, the training model for classification may be a training model obtained by training using training data, the training data including a plurality of images including a low image quality image as input data and a high image quality image obtained by a training model for image quality enhancement, and data labeled (annotated) with an image type as true value data. Here, the estimated (predicted) image type of the input data may be displayed together with information indicating the likelihood of each image type being included in the true value data during training (such as a numerical value indicating a ratio). In addition to the aforementioned images, the input data of the training model for classification may also include a high image quality image in which contrast is enhanced and noise is reduced by superposition processing of a plurality of low image quality images (e.g., averaging processing of a plurality of aligned low image quality images), etc. For example, the training model for additional training may be evaluated by comparing a plurality of high image quality images obtained from the same image using a training model for additional training and a training model (backup training model) that has not been additionally trained, or by comparing the analysis results of a plurality of high image quality images. Here, for example, it is possible to determine whether a comparison result of multiple high-image quality images (an example of changes due to additional training) or a comparison result of analysis results of multiple high-image quality images (an example of changes due to additional training) falls within a predetermined range, and the determination result can be displayed.

[0148] A training model obtained by performing training for each imaging area can be selectively used. Specifically, the information processing device may include a selection unit for selecting one of a plurality of training models, the plurality of training models including a first training model obtained by training with training data including a first imaging area (such as a lung or an eye to be examined) and a second training model obtained by training with training data including a second imaging area different from the first imaging area. Here, the information processing device may include a control unit (not shown) for performing additional training on the selected training model. The control unit may search for data including an imaging area corresponding to the selected training model and a captured image of the imaging area as paired data, and additionally train the selected training model using the searched data as training data based on the inspector's instructions. The imaging area corresponding to the selected training model may be obtained from the header information about the data or manually input by the inspector. For example, data may be searched in a server in a hospital or an external institution such as a laboratory via a network. Therefore, by using a captured image of the imaging area corresponding to the trained model, additional training for each imaging area can be effectively performed.

[0149] The selection unit and the control unit may be composed of a software module executed by a processor of the information processing device, such as a CPU or a microprocessing unit (MPU). The selection unit and the control unit may be composed of a circuit for providing a specific function, such as an application specific integrated circuit (ASIC), an independent device, etc.

[0150] If training data for additional training is obtained from a server in a hospital or an external institution such as a laboratory via a network, it is expected to reduce reliability degradation due to tampering, system failure during additional training, etc. To this end, the validity of the training data for additional training can be detected by a consistency check using a digital signature or hash. This can protect the training data for additional training. If the validity of the training data for additional training is not successfully detected as a result of the consistency check using a digital signature or hash, the information processing device gives a warning and does not perform additional training using the training data. The installation location of the server is not limited. The server can have any configuration, such as a cloud server, a fog server, or an edge server.

[0151] The aforementioned protection of data by consistency check is not limited to training data for additional training, and can be applied to data including medical images. The image management system can be configured to manage transactions of data including medical images between servers in multiple institutions through a distributed network. The image management system can be configured so that multiple blocks are connected in time series, and each block records a transaction history with a hash value of the previous block. Cryptography (such as lattice-based cryptography and quantum cryptography based on quantum key delivery) that is difficult to calculate even with quantum gates or other quantum computers can be used as a technology for performing consistency checks, etc. Here, the image management system can be a device and system for receiving and storing images captured by imaging devices and image-processed images. The image management system can also transmit images based on requests from connected devices, perform image processing on stored images, and request image processing from other devices. Examples of image management systems may include picture archiving and communication systems (PACS). Specifically, the image management system includes a database that can store received images and various types of information including associated patient information and imaging time. The image management system is connected to a network and can send and receive images, convert images, and send and receive various types of information associated with stored images based on requests from other devices.

[0152] (Sixth Modification)

[0153] In the aforementioned various exemplary embodiments and variations, in addition to manual instructions (for example, instructions given from a user interface, etc.), the inspector's instructions may also be instructions given by voice, etc. Here, for example, a machine learning engine (machine learning model) including a voice recognition engine (voice recognition model, training model for voice recognition) obtained by machine learning may be used. Manual instructions may be instructions given by character input, etc. using a keyboard, a touch panel, etc. Here, for example, a machine learning engine including a character recognition engine (character recognition model, training model for character recognition) obtained by machine learning may be used. The inspector's instructions may also be instructions using gestures. Here, a machine learning engine including a gesture recognition engine (gesture recognition model, training model for gesture recognition) obtained by machine learning may be used.

[0154] The inspector's instruction may be given as a detection result of the inspector's line of sight on the display screen (monitor) of the display unit. For example, the detection result of the line of sight may be a pupil detection result using a moving image of the inspector captured near the display screen (monitor) of the display unit. Here, the pupil in the moving image may be detected by using the aforementioned object recognition engine. In addition, the inspector's instruction may be an instruction based on brain waves, weak electrical signals flowing through the body, and the like.

[0155] In this case, for example, the training data may include character data, voice data (waveform data), etc., which indicate instructions for displaying the processing results of the aforementioned various training models as input data and execution commands for actually displaying the results of the processing of various training models, etc. on the display unit as true value data. For example, the training data may include character data, voice data, etc., which indicate instructions for displaying a high-image quality image obtained by a training model for image quality enhancement as input data and execution commands for displaying a high-image quality image as true value data and execution commands for activating a button for giving instructions to display a high-image quality image. It should be understood that, for example, the training data may be any training data in which the content of the instruction indicated by character data, voice data, etc. and the content of (one or more) execution commands correspond to each other. The voice data may be converted into character data by using an acoustic model, a language model, etc. The processing of reducing the noise data superimposed on the voice data using the waveform data obtained by multiple microphones may be performed. The information processing device may be configured so that characters, voices or other instructions and instructions given using a mouse, a touch panel, etc. may be selected based on the instructions of the inspector. The information processing apparatus may be configured so that whether to turn on or off characters, voice, or other instructions can be selected based on the examiner's instructions.

[0156] Here, for example, machine learning includes the aforementioned deep learning, and a recurrent neural network (RNN) may be used for at least a portion of a multi-stage neural network. Fig.9A and Fig. 9B As an example of a machine learning engine related to this modification, an RNN as a neural network for processing time series information is described. Fig. 10A and Fig. 10B Describe the Long Short-Term Memory (LSTM) network as a type of RNN.

[0157] Fig.9A The structure of RNN as a machine learning engine is shown. RNN 3520 has a ring network structure and inputs data x at time t t 3510 and output data h t 3530. Since the RNN 3520 has a ring network function and can pass the state of the current time to the next state, the RNN 3520 can process time series information. Fig. 9B An example of the input and output of a parameter vector at time t is shown. Data x t 3510 includes N pieces of data (Params1 to ParamsN). The data output from RNN 3520 is t3530 includes N pieces of data (Params1 to ParamsN) corresponding to the input data.

[0158] However, since RNN cannot process long-term information during back-propagation, LSTM can sometimes be used. LSTM includes a forget gate, an input gate, and an output gate, so it can learn long-term information. Fig. 10A The structure of LSTM is shown. The information to be passed from the network or LSTM 3540 to the next time t is the internal state c of the network called a cell. t-1 , and output data h t -1 . Lowercase letters (c, h, and x) in the figures represent vectors.

[0159] Next, Fig. 10B The details of LSTM 3540 are shown. Fig. 10B In the example, the forget gate network FG, the input gate network IG, and the output gate network OG are all sigmoid layers. Therefore, the forget gate network FG, the input gate network IG, and the output gate network OG all output vectors whose elements have values ​​from 0 to 1. The forget gate network FG determines how much past information to save. The input gate network IG determines which values ​​to update. The cell update candidate network CU is an activation function tanh layer. The cell update candidate network CU generates a new candidate vector to be added to the cell. The output gate network OG selects one or more elements of the cell candidate and selects the amount of information to be passed to the next time.

[0160] The aforementioned LSTM model is a basic form, and the network described here is not restrictive. The connections between networks can be changed. A quasi-recurrent neural network (QRNN) can be used instead of LSTM. In addition, the machine learning engine is not limited to neural networks, and boosting, support vector machines, etc. can be used. If the inspector's instructions are input by using characters, voices, etc., techniques related to natural language processing (such as sequence to sequence) can be applied. As a technology related to natural language processing, for example, a model that outputs for each input sentence can be applied. The aforementioned various training models are not limited to instructions applied to the inspector, and can be applied to outputs to the inspector. An interactive engine (interaction model, training model for interaction) that responds to the inspector using characters, voices, or other outputs can be applied.

[0161] As a technology related to natural language processing, a training model obtained by performing unsupervised learning using document data in advance can be used. As a technology related to natural language processing, a training model obtained by further performing transfer learning (or fine-tuning) on ​​the training model obtained by pre-learning depending on the intended use can be used. For example, as a technology related to natural language processing, a bidirectional encoder representation from a transformer (BERT) can be applied. As a technology related to natural language processing, a model that can extract (express) context (feature quantity) by predicting a specific word in a text from both the previous and the latter contexts can be applied. As a technology related to natural language processing, a model that can determine the relationship (continuity) between two sequences (sentences) in input time series data can be applied. As a technology related to natural language processing, a model that uses a transformer encoder in a hidden layer and inputs and outputs a vector sequence can be applied.

[0162] Here, the instructions of the inspector to which this variant can be applied can be any one of the instructions related to the following: switching of the display of various images and analysis results, selection of a depth range for generating an en-face image, selection of training data for additional training, selection of a training model, and output (e.g., display and transmission), storage, etc. of the results obtained by using the various training models described in the aforementioned various exemplary embodiments and variants. The instructions of the inspector to which this variant can be applied are not limited to instructions after imaging, and can be instructions before imaging. Examples include instructions about various adjustments, instructions about the setting of various imaging conditions, and instructions about starting imaging. The instructions of the inspector to which this variant can be applied can be instructions about switching of display screens (screen transitions).

[0163] The machine learning model may be a model that combines an image-related machine learning model (such as a CNN) and a time series data-related machine learning model (such as an RNN). Such a machine learning model can learn, for example, the relationship between feature quantities related to an image and feature quantities related to time series data. For example, if the input layer of the machine learning model is a CNN and the output layer is an RNN, training data including medical images can be used as input data and text related to the medical images (e.g., the presence or absence of lesions, the type of lesions, recommendations for the next examination, etc.) can be used as output data to perform training. This enables, for example, even an inspector who does not have much medical experience to easily find out medical information about medical images, because the medical information related to medical images is automatically described in text form. If the input layer of the machine learning model is an RNN and the output layer is a CNN, training data including medical texts about lesions, findings, diagnoses, etc. can be used as input data and medical images corresponding to the medical texts can be used as output data to perform training. This enables, for example, inspectors to easily search for medical images related to cases to be observed.

[0164] A machine translation engine (machine translation model, training model for machine translation) for machine translation of characters, voices and other texts into a given language can be used for instructions from the inspector and output to the inspector. The information processing device can be configured so that a given language can be selected based on the inspector's instructions. For example, the aforementioned techniques related to natural language processing (e.g., sequence to sequence) can be applied to the machine translation engine. For example, the information processing device can be configured so that after the text input to the machine translation engine is machine translated, the machine translated text is input to a character recognition engine, etc. For example, the information processing device can be configured so that the text output from the aforementioned various training models is input to the machine translation engine, and the text output from the machine translation engine is output.

[0165] The aforementioned various training models can be used in combination. For example, the information processing device can be configured so that the characters corresponding to the inspector's instructions are input to the character recognition engine, and the voice obtained from the input characters is input to another type of machine learning engine (such as a machine translation engine). For example, the information processing device can be configured so that the characters output from another type of machine learning engine are input to the character recognition engine, and the voice obtained from the input characters is output. For example, the information processing device can be configured so that the voice corresponding to the inspector's instructions is input to the voice recognition engine, and the characters obtained from the input voice are input to another type of machine learning engine (such as a machine translation engine). For example, the information processing device can be configured so that the voice output from another type of machine learning engine is input to the voice recognition engine, and the characters obtained from the input voice are displayed on the display unit. Here, the information processing device can be configured so that it can be selected to output character output or voice output to the inspector based on the inspector's instructions. The information processing device can also be configured so that it can be selected to use character input or voice input as the inspector's instructions based on the inspector's instructions. The aforementioned various configurations can be adopted based on the selection made by the inspector's instructions.

[0166] (Seventh Modification)

[0167] In the aforementioned various exemplary embodiments and variations, a high image quality image or the like can be stored in a storage unit based on the inspector's instruction. When a file name is registered to store a high image quality image or the like after the inspector's instruction, a file name including the following information (e.g., characters) can be displayed as a recommended file name at a part of the file name (e.g., at the beginning or at the end) in an editable state based on the inspector's instruction, the information indicating that the image has been generated by processing using a training model for image quality enhancement (image quality enhancement processing). When a high image quality image is displayed on various display screens such as a report screen on a display unit, an indication that the displayed image is a high image quality image generated by processing using a training model for image quality enhancement can be displayed together with the high image quality image. In this case, the user can easily recognize from the indication that the displayed high image quality image is not the image itself obtained by imaging. This can reduce misdiagnosis and improve diagnostic efficiency. The indication of a high image quality image generated by processing using a training model for image quality enhancement can be an indication of any mode from which an input image and a high image quality image generated by processing can be identified. Not only can the results of processing using a training model for image quality enhancement be displayed, but the results of processing using the various training models described above can also be displayed with an indication that the results have been generated by processing using these types of training models.

[0168] Here, a display screen such as a report screen can be stored as image data in a storage unit based on the inspector's instructions. For example, a report screen can be stored in a storage unit as an image in which high image quality images, etc. are arranged side by side and the image is an indication of a high image quality image generated by processing using a training model for image quality enhancement. As an indication of a high image quality image generated by processing using a training model for image quality enhancement, an indication of which training data the training model for image quality enhancement was trained with can be displayed on the display unit. Such an indication may include a description of the types of input data and true value data in the training data, and any indication related to the input data and the true value data, such as an imaging area included in the input data and the true value data. Here, not only for processing using a training model for image quality enhancement, but also for processing using the aforementioned various training models, an indication of which training data the training model of this type was trained with can be displayed on the display unit.

[0169] The information processing device may be configured so that information (e.g., characters) indicating that an image has been generated by processing using a training model for image quality enhancement is displayed or stored in a state superimposed on a high image quality image or the like. Here, the superimposition position on the image may be located in any area (e.g., in a corner of the image) that does not overlap with an area where an area of ​​interest to be imaged or the like is displayed. The information processing device may determine such a non-overlapping area and superimpose the information on the determined area.

[0170] The information processing device may be configured such that: if the image quality enhancement button is set to be activated (image quality enhancement processing is enabled) on the initial display screen of the report screen by default, a report image corresponding to the report screen including a high image quality image, etc. is transmitted to a server such as an external storage unit based on the inspector's instruction. The information processing device may be configured such that: if the image quality enhancement button is set to be activated by default, a report image corresponding to the report screen including a high image quality image, etc. is (automatically) transmitted to the server at the end of the examination (for example, when the imaging observation screen or the preview screen is switched to the report screen based on the inspector's instruction). Here, the information processing device may be configured so that a report image generated based on various default settings (for example, settings related to at least one of the following items: a depth range for generating an en-face image on the initial display screen of the report screen, the presence or absence of an overlay analysis map, whether the image is a high image quality image, and whether the screen is a display screen for follow-up) is transmitted to the server.

[0171] (Eighth Modification)

[0172] In the aforementioned various exemplary embodiments and modifications, an image (e.g., a high image quality image, an image indicating an analysis result such as an analysis map, an image indicating an object recognition result, or an image indicating a segmentation result) obtained by a first type of training model among the aforementioned various training models may be input to a second type of training model different from the first type. Here, the information processing device may be configured to generate a result of processing by the second type of training model (such as an analysis result, a diagnosis result, an object recognition result, or a segmentation result).

[0173] By using the result of the processing of the first type training model (such as an analysis result, a diagnosis result, an object recognition result, or a segmentation result), an image to be input to a second type training model different from the first type training model among the aforementioned various training models can be generated from the image input to the first type training model. Here, the generated image may be suitable as an image for processing by the second type training model. Therefore, the accuracy of the image (for example, a high image quality image, an image indicating an analysis result such as an analysis map, an image indicating an object recognition result, or an image indicating a segmentation result) obtained by inputting the generated image into the second type training model can be improved. The information processing device can be configured to input a common image into the first type training model and the second type training model to generate (or display) the processing result using the training model. Here, for example, the information processing device can be configured to simultaneously (linkedly) generate (or display) the processing result using the training model based on the inspector's instructions. The information processing device may be configured so that the type of image to be input (e.g., high image quality image, object recognition result, segmentation result, or similar case image), the type of processing result to be generated (or displayed) (e.g., high image quality image, diagnosis result, analysis result, object recognition result, segmentation result, or similar case image), input type, and output type (such as character, voice, and language) can be selected based on the inspector's instructions. Here, the information processing device may be configured so that at least one training model is selected depending on the selected type. If multiple training models are selected here, the combination of multiple training models (such as the order of data input) can be determined based on the selected type. For example, the information processing device may be configured so that the type of image to be input and the type of processing result to be generated (or displayed) can be selected to be different. If the same type is selected, the information processing device may be configured to output information to the inspector for prompting different selections. The training model can be executed at any location. For example, some of the multiple training models can be used by a cloud server, while other training models can be configured to be used by different servers such as a fog server or an edge server.

[0174] The various training models mentioned above may be models obtained by training using training data including two-dimensional medical images of test subjects, or may be models obtained by training using training data including three-dimensional medical images of test subjects.

[0175] Similar case image search using an external database stored in a server or the like can be performed with analysis results, diagnosis results, etc. processed by the aforementioned various training models as search keywords. Similar case image search using an external database stored in a server or the like can also be performed with object recognition results, segmentation results, etc. processed by the aforementioned various training models as search keywords. In cases such as when a plurality of images stored in a database are managed using feature quantities of corresponding plurality of images that have been attached as attachment information by machine learning or the like, a similar case image search engine (similar case image search model, training model for similar case image search) can be used, which uses the image itself as a search keyword. For example, an information processing device can search for similar case images related to a medical image in various medical images by using a training model for similar case image search (different from a training model for image quality enhancement). For example, the display control unit 103 can display similar case images obtained from various medical images on a display unit by using a training model for similar case image search. Here, for example, a similar case image is an image having a feature quantity similar to that of a medical image input to a training model. For example, if a medical image input to the training model includes a partial area such as an abnormal area, the similar case image is an image having a feature quantity similar to that of the partial area such as the abnormal area. For example, this not only enables effective training of precise search of similar case images, but also enables the examiner to efficiently diagnose the abnormal area if the medical image includes an abnormal area. In addition, multiple similar case images can be searched, and the multiple similar case images can be displayed in a manner in which the order of similarity of the feature quantity is identifiable. The information processing device can be configured to additionally train a training model for similar case image search by using training data, the training data including images selected from multiple similar case images based on the examiner's instructions and the feature quantity of the image.

[0176] (Ninth Modification)

[0177] In the aforementioned various exemplary embodiments and variations, the processing for generating motion contrast data is not limited to a configuration in which processing is performed based on the brightness value of a tomographic image. Various types of processing can be applied to interference signals obtained by an optical coherence tomography (OCT), interference signals to which Fourier transform is applied, signals to which given processing is applied, and tomographic data including tomographic images, etc. based on these signals. A similar effect can be provided even in this case. For example, when an optical fiber system using an optical coupler is used as a beam splitting unit, a spatial optical system including a collimator and a beam splitter can be used. The OCT can be configured so that some components included in the OCT are separated from the OCT. A Michelson interferometer configuration can be used for an OCT interference optical system. A Mach-Zehnder interferometer configuration can be used. The OCT can be a spectral domain OCT (SD-OCT) using a superluminescent diode (SLD) as a light source. The OCT can be any other type of OCT, such as a swept source OCT (SS-OCT) using a wavelength swept light source capable of sweeping the wavelength of the emitted light. In addition, the present invention can also be applied to a line-OCT device (or SS-line-OCT device) using line light. The present invention can also be applied to a full-field-OCT device (or SS-full-field-OCT device) using surface light. Although the information processing device obtains interference signals obtained by OCT and three-dimensional tomographic imaging images generated by the information processing device, etc., the configuration of the information processing device to obtain such signals and images is not limited thereto. For example, the information processing device can obtain such signals and images from a server or imaging device connected via a local area network (LAN), a wide area network (WAN), the Internet, etc.

[0178] The training model may be located in the information processing device. For example, the training model may be composed of a software module executed by a processor such as a CPU. Alternatively, the training model may be located in other servers connected to the information processing device, etc. In this case, the information processing device may perform image quality enhancement processing using the training model by connecting to the server including the training model via a given network such as the Internet.

[0179] (Tenth Modification)

[0180] The medical images to be processed by the information processing apparatus (medical image processing apparatus) or the information processing method (medical image processing method) according to the aforementioned various exemplary embodiments and modifications may include images acquired by using any modality (imaging device, imaging method). The medical images to be processed may include medical images acquired by a given imaging device or the like, and images generated by the medical image processing apparatus or the medical image processing method.

[0181] The image to be processed also includes an image of a predetermined area of ​​the subject (test subject), and the image of the predetermined area includes at least a portion of the predetermined area of ​​the subject. The medical image may include other areas of the subject. The medical image may be a still image or a moving image, and may be a monochrome image or a color image. The medical image may be an image showing the structure (shape) of the predetermined area or an image showing its function. Examples of images showing functions include images showing hemodynamics (blood flow, blood flow rate, etc.), such as OCTA images, Doppler OCT images, functional magnetic resonance imaging (fMRI) images, and ultrasound Doppler images. The predetermined area of ​​the subject can be determined based on the imaging target, and may include organs such as human eyes (eyes to be examined), brains, lungs, intestines, hearts, pancreas, kidneys, and livers, as well as areas such as the head, chest, legs, and arms.

[0182] The medical image may be a tomographic image or a frontal image of the subject. Examples of frontal images include frontal fundus images, frontal images of the anterior part of the eye, fundus images obtained by fluorescence imaging, and en-face images generated using data of at least a portion of the data range (three-dimensional OCT data) obtained by OCT in the depth direction of the imaging target. The en-face image may be an OCTA en-face image (motion contrast frontal image) generated using at least a portion of the range of three-dimensional OCTA data (three-dimensional motion contrast data) in the depth direction of the imaging target. Three-dimensional OCT data and three-dimensional motion contrast data are examples of three-dimensional medical image data.

[0183] As used herein, motion contrast data refers to data indicating changes between multiple pieces of volume data obtained by controlling the measuring light to scan the same area (same position) of the eye to be examined multiple times. Here, the volume data includes multiple tomographic images obtained at different positions. Motion contrast data can be obtained as volume data by obtaining data indicating changes between multiple tomographic images obtained at substantially the same position at each different position. Motion contrast en-face images are also referred to as OCTA en-face images (OCTA en-face images) associated with OCT angiography (OCTA) for measuring blood flow motion. Motion contrast data is also referred to as OCTA data. For example, motion contrast data can be determined as a solution to divide a correlation value, a variance value, or a maximum value by a minimum value (maximum value / minimum value) between two tomographic images or corresponding interference signals, and motion contrast data can be determined by any conventional method. Here, for example, two tomographic images can be obtained by controlling the measuring light to scan the same area (same position) of the eye to be examined multiple times.

[0184] For example, an en-face image is a frontal image generated by projecting data in a range between two layer boundaries on an XY plane. Here, a frontal image is generated by projecting data corresponding to a depth range onto a two-dimensional plane or by integrating the data, and the depth range is at least a part of the depth range of volume data (three-dimensional tomographic image) obtained using optical interference and is determined based on two reference planes. An en-face image is a frontal image generated by projecting data corresponding to a depth range determined based on a retinal layer detected in the volume data onto a two-dimensional plane. For example, data corresponding to a depth range determined based on two reference planes can be projected onto a two-dimensional plane, for example, by using a technique for using a representative value of data in a depth range as a pixel value on a two-dimensional plane. Here, the representative value may include values ​​such as the average, median, and maximum of pixel values ​​in a depth range of an area surrounded by two reference planes. An example of a depth range associated with an en-face image may be a range that includes a predetermined number of pixels in a deeper direction or a shallower direction than any of the two layer boundaries associated with the detected retinal layer. An example of the depth range associated with the en-face image may be a range obtained by modifying (shifting) the range between two-layer boundaries associated with the detected retinal layers based on an instruction of the operator.

[0185] An imaging device is a device for capturing images for diagnosis. Examples of imaging devices include devices that obtain an image of a predetermined area of ​​a subject by irradiating the predetermined area with light, radiation such as X-rays, electromagnetic waves, ultrasound, etc., and devices that obtain an image of a predetermined area by detecting radiation emitted from an object. More specifically, the imaging devices according to the aforementioned various exemplary embodiments and modifications include at least an X-ray imaging device, a computed tomography (CT) device, a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, a single photon emission computed tomography (SPECT) device, an SLO device, an OCT device, an OCTA device, a fundus camera, and / or an endoscope.

[0186] The OCT device may include a time domain OCT (TD-OCT) device and a Fourier domain OCT (FD-OCT) device. The Fourier domain OCT device may include a spectral domain OCT (SD-OCT) device and a swept source OCT (SS-OCT) device. The OCT device may also include a Doppler-OCT device. The SLO device and the OCT device may include an adaptive optical SLO (AO-SLO) device and an adaptive optical OCT (AO-OCT) device using an adaptive optical system. The SLO device and the OCT device may also include a polarization-sensitive SLO (PS-SLO) device and a polarization-sensitive OCT (PS-OCT) device for visualizing information about polarization phase difference or depolarization. The SLO device and the OCT device may include a pathological microscope SLO device and a pathological microscope OCT device. The SLO device and the OCT device may include a handheld SLO device and a handheld OCT device. The SLO device and the OCT device may include a catheter SLO device and a catheter OCT device.

[0187] (Other Exemplary Embodiments)

[0188] The present invention is also implemented by performing the following processing. That is, the processing includes providing software (program) for implementing one or more functions of the various exemplary embodiments and variants described above to a system or device via a network or various storage media, and reading and executing the program by a computer (or CPU, MPU, etc.) of the system or device.

[0189] The present invention can also be implemented by providing software (program) for implementing one or more functions of the aforementioned various exemplary embodiments and variants to a system or device via a network or various storage media, and the program is read and executed by a computer of the system or device. A computer includes one or more processors or circuits, and may include a network of multiple individual computers or multiple individual processors or circuits to read and execute computer executable instructions.

[0190] Here, the processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field programmable gate array (FPGA). The processor or circuit may also include a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0191] The present invention is not limited to the above embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to make the public aware of the scope of the present invention, the following claims are made.

[0192] This application claims priority from Japanese Patent Application No. 2019-068893 filed on March 29, 2019, No. 2019-183352 filed on October 3, 2019, and No. 2019-220765 filed on December 5, 2019, which are hereby incorporated by reference herein.

Claims

1. An information processing device, comprising: a storage unit configured to store information individually set for each of optical coherence tomography (OCT) imaging and fundus imaging as a transmission setting for a plurality of pieces of imaging data acquired by each of the OCT imaging and the fundus imaging; and A transmission unit is configured to transmit imaging data obtained through one of the OCT imaging and the fundus imaging based on stored information individually set for the one of the OCT imaging and the fundus imaging, by switching from an imaging screen of the one of the OCT imaging and the fundus imaging to a display screen other than an imaging screen of the other of the OCT imaging and the fundus imaging as a trigger for starting transmission after one of the OCT imaging and the fundus imaging is completed.

2. The information processing device according to claim 1, wherein: The transmission unit is configured to: when the following information is stored and as the initial display setting of the report screen, the image quality enhancement processing is enabled, the information, i.e., the transmission setting for a type of imaging sets the transmission of the report image to be enabled, transmits the following report image as imaging data, the report image corresponding to the report screen displaying the second medical image obtained by performing image quality enhancement processing on the first medical image obtained by the one type of imaging.

3. The information processing device according to claim 2, wherein: The image quality enhancement process is a process for generating a second medical image from a first medical image by using a training model obtained by training using medical images of a test subject.

4. The information processing device according to claim 2 or 3, wherein: The transmission setting is configured such that a transmission setting for data including the first medical image and the second medical image as a set is includeable.

5. The information processing device according to claim 4, wherein: The data comprising the set is training data for additional training.

6. The information processing device according to claim 1, wherein: A plurality of ophthalmic imaging apparatuses configured to perform a plurality of different types of imaging on an eye to be examined of a test subject include a fundus camera and an optical coherence tomography (OCT) apparatus.

7. The information processing device according to claim 1, in, The transmission setting is configured so that a plurality of modes are registerable, and The transmission unit is configured to transmit imaging data of the test subject acquired by one type of imaging based on the stored information in the order of the plurality of registration patterns.

8. The information processing device according to claim 1, wherein: The transmission setting is configured to be able to individually set a corresponding plurality of different types of imaging based on an instruction from an examiner.

9. The information processing device according to claim 1, wherein: The transfer settings include the transfer contents, transfer type, and transfer destination as common settings, and the image size and the presence or absence of automatic transfer as individual settings.

10. The information processing device according to claim 1, further comprising: A display control unit configured to display the imaging data on the display unit.

11. An information processing method, comprising: storing information individually set for each of OCT imaging and fundus imaging as transmission settings for a plurality of pieces of imaging data acquired by each of OCT imaging and fundus imaging; and After one of OCT imaging and fundus imaging is completed, the imaging data obtained by the one of OCT imaging and fundus imaging is transmitted based on the stored information set separately for the one of OCT imaging and fundus imaging, with switching from the imaging screen of the one of OCT imaging and fundus imaging to other display screens other than the imaging screen of the other of OCT imaging and fundus imaging as a trigger for starting transmission.

12. A computer-readable storage medium storing a program for causing a computer to execute each step of the information processing method according to claim 11.

Images