Inspection support device, inspection support method, and inspection support program

Abstract

To provide an examination support device, etc. for appropriately diagnosing an object spot in which a doctor, who is a user, takes interest of an image captured by an endoscope.SOLUTION: An examination support device includes: an acquisition part for sequentially acquiring images captured with a camera unit inserted into the body of a subject; a diagnosis part for diagnosing the inside of the body on the basis of a set partial region of the captured image; and a display control part for sequentially displaying, in a display part, captured images with a region wider than the partial region as a display region in a mode that the user can recognize the partial region. When the partial region is displayed in the display part in this manner, the user has only to operate the camera unit so that an object spot to be diagnosed falls within the region, thereby achieving both ease of operation and certainty of diagnosis.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an inspection support device, an inspection support method, and an inspection support program. [Background technology]

[0002] There is known an endoscopic system that uses an endoscope to capture images of, for example, the inside of a subject's stomach and displays the images on a monitor. Recently, examination support devices that analyze images captured by the endoscopic system and notify doctors of the results have become popular (see, for example, Patent Document 1). In addition, a usage mode in which an examination support device is connected to an endoscopic system and the in-vivo images captured by the endoscopic system are analyzed by the examination support device in almost real time is also approaching the practical stage. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-42156 Summary of the Invention [Problem to be solved by the invention]

[0004] When diagnosing the presence or absence of a lesion in the entire image area of ​​an acquired image, the examination support device may output a series of erroneous diagnostic results due to artifacts caused by multiple lesions, contamination by body fluids, lighting, etc. Outputting and displaying diagnostic results for multiple locations on a single image, including such erroneous diagnostic results, can clutter the display and impair the visibility of the examining physician. On the other hand, displaying only diagnostic results with a high probability of lesions can result in erroneous diagnostic results, potentially resulting in the diagnosis of a lesion that should be recognized being overlooked. When physicians find a suspicious area while observing real-time images, they often want the examination support device to diagnose only that area. However, adjusting the field of view of the camera unit so that only that area is captured across the entire image area can significantly impair workability or may even be physically impossible.

[0005] The present invention has been made to solve such problems, and provides an examination support device etc. that appropriately diagnoses target areas of interest to the user, i.e., a doctor, in images captured by an endoscope. [Means for solving the problem]

[0006] The examination assistance device in a first aspect of the present invention includes an acquisition unit that sequentially acquires captured images taken by a camera unit inserted into the body of a subject, a diagnosis unit that performs a diagnosis inside the body based on a set partial area of ​​the captured image, and a display control unit that sequentially displays the captured images on a display unit as a display area that is larger than the partial area in a manner that allows a user to recognize the partial area.

[0007] An examination assistance method in a second aspect of the present invention is an examination assistance method that uses an examination assistance device that has an acquisition unit, a display control unit, and a diagnosis unit, and includes an acquisition step in which the acquisition unit sequentially acquires captured images taken by a camera unit inserted into the subject's body, a display step in which the display control unit sequentially displays the captured images on the display unit as a display area an area larger than a set partial area of ​​the captured image in a manner that allows a user to recognize the set partial area of ​​the captured image, and a diagnosis step in which the diagnosis unit performs a diagnosis of the inside of the body based on the set partial area of ​​the captured image.

[0008] The examination assistance program in a third aspect of the present invention is an examination assistance program that controls an examination assistance device that includes an acquisition unit, a display control unit, and a diagnosis unit, and causes a computer to execute an acquisition step that causes the acquisition unit to sequentially acquire captured images taken by a camera unit inserted into the body of a subject, a display step that causes the display control unit to sequentially display the captured images on the display unit as a display area an area larger than a set partial area of ​​the captured image in a manner that allows a user to recognize the set partial area of ​​the captured image, and a diagnosis step that causes the diagnosis unit to perform a diagnosis of the inside of the body based on the set partial area of ​​the captured image.

[0009] An examination assistance device in a fourth aspect of the present invention includes an acquisition unit that sequentially acquires captured images taken by a camera unit inserted into the body of a subject, a display control unit that sequentially displays the captured images on a display unit as a display area that is larger than a set partial area of ​​the captured image in a manner that allows a user to recognize the partial area, and a diagnosis unit that performs a diagnosis inside the body based on the captured images and outputs the diagnosis results for the partial area separately from the diagnosis results for other areas.

[0010] An examination assistance method in a fifth aspect of the present invention is a method of examination assistance using an examination assistance device having an acquisition unit, a display control unit, and a diagnosis unit, and includes an acquisition step in which the acquisition unit sequentially acquires captured images taken by a camera unit inserted into the body of a subject, a display step in which the display control unit sequentially displays the captured images on the display unit as a display area an area larger than a set partial area of ​​the captured image in a manner that allows a user to recognize the partial area, and a diagnosis step in which the diagnosis unit performs a diagnosis of the inside of the body based on the captured images and outputs the diagnosis result for the partial area separately from the diagnosis result for other areas.

[0011] An examination assistance program in a sixth aspect of the present invention is an examination assistance program that controls an examination assistance device having an acquisition unit, a display control unit, and a diagnosis unit, and causes a computer to execute an acquisition step that causes the acquisition unit to sequentially acquire captured images taken by a camera unit inserted into the body of a subject; a display step that causes the display control unit to sequentially display the captured images on the display unit as a display area an area larger than a set partial area of ​​the captured image in a manner that allows a user to recognize the partial area; and a diagnosis step that causes the diagnosis unit to perform a diagnosis inside the body based on the captured images and output the diagnostic results for the partial area separately from the diagnostic results for other areas. [Effects of the Invention]

[0012] The present invention can provide an examination support device or the like that can appropriately diagnose a target area of ​​interest to a doctor who is the user, from an image captured by an endoscope. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram showing an endoscopic examination using an endoscope system and an examination support device according to this embodiment. [Figure 2] FIG. 2 is a hardware configuration diagram of the inspection support device. [Figure 3] 10A and 10B are diagrams showing how the display screen changes in response to the doctor's operation of the camera unit. [Figure 4] FIG. 10 is a diagram illustrating the process up to displaying the diagnosis result. [Figure 5] 10A and 10B are diagrams illustrating a procedure for determining a diagnostic target image from a change in a captured image. [Figure 6] FIG. 4 is a flowchart illustrating a processing procedure of a calculation processing unit. [Figure 7] 10A and 10B are diagrams illustrating step-by-step diagnostic processing of a divided area in a first modified example. [Figure 8] FIG. 10 is a hardware configuration diagram of an inspection support device according to a second modified example. [Figure 9] 10 is an example of a user interface screen relating to setting of a diagnostic region in the second modified example. [Figure 10] FIG. 10 is a diagram illustrating the process up to displaying the diagnosis result in the third modified example. [Figure 11] FIG. 11 is a flowchart illustrating a processing procedure of a calculation processing unit in a third modified example. DETAILED DESCRIPTION OF THE INVENTION

[0014] The present invention will be described below through embodiments of the invention, but the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems.

[0015] FIG. 1 is a diagram showing an endoscopic examination using an endoscopic system 200 and an examination support device 100 according to this embodiment. Both the endoscopic system 200 and the examination support device 100 are installed in an examination space. The endoscopic system 200 includes a camera unit 210, which is inserted from the oral cavity of a lying subject into an internal organ such as the stomach, and transmits image signals of images captured of the inside of the organ to the system main body, as shown in the figure. The insertion of the camera unit 210 into the organ and the image capturing operation are performed by a doctor.

[0016] The endoscope system 200 includes a system monitor 220 configured by, for example, a liquid crystal panel, and processes image signals sent from the camera unit 210 to display a visible captured image 221 on the system monitor 220. The endoscope system 200 also displays examination information 222, including subject information and camera information of the camera unit 210, on the system monitor 220.

[0017] The examination support device 100 is connected to the endoscope system 200 via a connection cable 250. The endoscope system 200 also transmits a display signal to be sent to the system monitor 220 to the examination support device 100 via the connection cable 250. That is, the display signal in this embodiment is an example of an image signal provided by the endoscope system 200 to an external device. The examination support device 100 includes a display monitor 120 configured by, for example, a liquid crystal panel as a display unit, and extracts an image signal corresponding to a captured image 221 from the display signal sent from the endoscope system 200 and sequentially displays the extracted image on the display monitor 120 as a visible captured image 121.

[0018] The doctor proceeds with the examination while visually checking the captured image 121 that is displayed almost in real time in response to the operation of the camera unit 210. As will be described in detail later, a diagnostic area frame 121a is superimposed on the captured image 121 that is displayed almost in real time, and the doctor operates the camera unit 210 so that the target area that the doctor wants the examination support device to diagnose falls within the diagnostic area frame 121a.

[0019] As will be described later, when the test support device 100 detects a diagnostic instruction, it generates image data of a partial region of the captured image 121 that corresponds to the diagnostic region frame 121a. Then, it performs a diagnosis based on the image data and displays the resulting diagnostic information 122 on the display monitor 120. Note that Fig. 1 shows a state in which the captured image 121, which is generated sequentially from a display signal in almost real time, is displayed in a situation in which a diagnostic instruction has not been detected, and the diagnostic information 122 indicates that the device is in a diagnosis standby state.

[0020] 2 is a hardware configuration diagram of the test support device 100. The test support device 100 is mainly composed of an arithmetic processing unit 110, a display monitor 120, an input / output interface 130, an input device 140, and a storage unit 150. The arithmetic processing unit 110 is a processor (CPU: Central Processing Unit) that controls the test support device 100 and executes programs. The processor may be configured to work in conjunction with an arithmetic processing chip such as an ASIC (Application Specific Integrated Circuit) or a GPU (Graphics Processing Unit). The arithmetic processing unit 110 reads out the test support program stored in the storage unit 150 and executes various processes related to test support.

[0021] As described above, the display monitor 120 is a monitor equipped with, for example, a liquid crystal panel, and visibly displays the captured image 121 and the diagnostic information 122. The input / output interface 130 is a connection interface for exchanging information with an external device, including a connector for connecting the connection cable 250. The input / output interface 130 includes, for example, a LAN unit, and imports an inspection assistance program and update data for the analytical neural network 151 (described later) from the external device and transfers them to the arithmetic processing unit 110.

[0022] The input device 140 is, for example, a keyboard, a mouse, or a touch panel superimposed on the display monitor 120, and doctors and assistants operate these to change the settings of the examination support device 100 or input information necessary for the examination.

[0023] The storage unit 150 is a non-volatile storage medium, and is configured, for example, by an HDD (Hard Disk Drive). The storage unit 150 can store various parameter values, functions, display element data, lookup tables, and the like used for control and calculation, in addition to programs for controlling and executing processes of the examination support device 100. The storage unit 150 particularly stores an analytical neural network 151. The analytical neural network 151 is a trained model that, when inputted, calculates the probability that a lesion is present in the image captured by the camera unit 210. Note that the storage unit 150 may be configured with multiple pieces of hardware; for example, a storage medium for storing the program and a storage medium for storing the analytical neural network 151 may be configured with separate pieces of hardware.

[0024] The arithmetic processing unit 110 also serves as a functional calculation unit that executes various calculations in accordance with processing instructed by the examination assistance program. The arithmetic processing unit 110 can function as an acquisition unit 111, a diagnosis unit 112, and a display control unit 113. The acquisition unit 111 sequentially acquires display signals sent from the endoscope system 200 and develops each into a signal reproduction image. Furthermore, the acquisition unit 111 determines an image area representing the captured image captured by the camera unit 210 by searching for a change area where the difference between the preceding and following signal reproduction images is equal to or greater than a reference amount. Once the acquisition unit 111 determines the image area, it generates a captured image 121 from the image area for display signals acquired thereafter.

[0025] The diagnosis unit 112 inputs a diagnostic region image, which is a preset partial region cut out from the captured image 121, into the analytical neural network 151 read from the storage unit 150, calculates the probability of the presence of a lesion, and generates diagnostic information. The display control unit 113 controls the display on the display monitor 120 by generating a display signal for a display screen to be displayed on the display monitor 120 and transmitting it to the display monitor 120. In a situation where a diagnostic instruction has not been detected, the display control unit 113 generates a display signal in which a diagnostic region frame 121a is superimposed on the captured image 121, which is successively updated and displayed.

[0026] 3A and 3B are examples of the display screen of the display monitor 120, showing how the screen changes as the doctor operates the camera unit 210. Both Fig. 3A and Fig. 3B show how the captured image 121, which is sequentially generated from the display signal output by the endoscope system 200, is displayed in a situation where no diagnostic instruction has been detected. In other words, the displayed captured image 121 is substantially the same as the image currently being captured by the camera unit 210 inserted into the organ.

[0027] The diagnosis unit 112 diagnoses a partial region set in the center of the captured image 121 as a diagnostic region. The display control unit 113 displays a diagnostic region frame 121a on the display monitor 120, superimposed on the captured image 121 so as to surround the diagnostic region, so that the user, i.e., a doctor, can recognize the diagnostic region. In this situation, the diagnosis information 122 indicates that the device is in a standby state for receiving a diagnostic instruction.

[0028] 3(a) shows an example of a display screen when a doctor finds an observation target 900, which is a suspected lesion, in a captured image 121 that is being observed. At this point, the observation target 900 is located on the periphery of the captured image 121. In addition, an artifact 910 has occurred in a corner of the captured image 121.

[0029] When the diagnosis unit 112 extracts lesion candidates from the entire image region of the captured image 121, there is a possibility that not only the observation target 900 but also the artifact 910 will be extracted. For example, if the analytical neural network 151 calculates estimated probabilities equal to or greater than the threshold for both the observation target 900 and the artifact 910, the probabilities will be displayed for both, cluttering the display screen and reducing visibility. Alternatively, if the probability is displayed only for the candidate with the highest calculated estimated probability, it is possible that only the erroneous result for the artifact 910 will be displayed. Furthermore, even if the extracted candidates are correct as lesion candidates, if there is a high probability that the candidate is not a candidate of interest to the doctor, the probability that the candidate is a lesion of interest will not be displayed.

[0030] Furthermore, the captured image 121 may contain artifacts 910 and other elements other than organ tissue. Therefore, inputting the entire image region of the captured image 121 into the analytical neural network 151 may result in an erroneous calculation result. Since the analytical neural network 151 is a trained model that has learned from correct images annotated with information indicating whether they represent diseased tissue or not, it tends to output erroneous calculation results for images other than organ tissue that it has not trained on. However, if a doctor finds a concerning observation target 900 within the entire image region, adjusting the angle of view of the camera unit 210 so that the observation target 900 expands to fill the entire image region significantly impairs workability. Furthermore, the doctor may lose track of the relative position of the observation target 900 within the target organ.

[0031] Therefore, as described above, the diagnosis unit 112 diagnoses only the diagnosis area set in the center of the captured image 121, and the display control unit 113 displays the diagnosis area frame 121a so as to surround the diagnosis area. When the doctor finds the observation target 900 in the periphery of the captured image 121 as shown in FIG. 3(a), he operates the camera unit 210 so that the observation target 900 fits into the diagnosis area frame 121a, as shown in FIG. 3(b). In this case, the operation can be performed quickly and easily by simply shifting the camera unit 210. Once the doctor has placed the observation target 900 within the diagnosis area frame 121a, he issues a diagnosis instruction to the diagnosis unit 112 to make a diagnosis.

[0032] In this way, when the diagnostic region is set in the center, it is easy for the doctor to grasp which part of the organ is being diagnosed. Furthermore, artifacts are less likely to occur in the center of the captured image 121. Even if artifacts do occur, if the doctor is aware of the artifacts, they can avoid overlapping artifacts by delaying the timing of diagnostic instructions or adjusting the angle of the camera unit 210. In this embodiment, the display control unit 113 displays the diagnostic region frame 121a superimposed on the entire image region of the captured image 121. However, as long as the display region includes the diagnostic region and is wider than the diagnostic region, the captured image 121 may be partially cut out. For example, if the entire image region of the captured image 121 is rectangular, the image may be displayed in a square or circular shape by cutting out a portion of the periphery.

[0033] In this embodiment, the diagnostic area frame 121a is displayed as a dotted line graphic as shown in the figure, but the display format is not limited to this. For example, a visual effect such as a colored frame or a blinking frame may be used. Furthermore, the present invention is not limited to the superimposition of a graphic frame, and any visual effect may be used as long as it allows the doctor to recognize the diagnostic area. For example, a visual effect may be used in which the diagnostic area in the captured image 121 is brightened and other areas are relatively darkened.

[0034] 4 is a diagram illustrating the processing from detection of a diagnostic instruction to display of a diagnostic result. A signal playback image 225 acquired by the acquisition unit 111 based on a display signal sent from the endoscope system 200 and displayed is the same as the display image displayed on the system monitor 220 of the endoscope system 200. The signal playback image 225 includes a captured image 221 and examination information 222. The examination information 222 is, for example, text information.

[0035] When it is determined as described above which image area of ​​the signal reproduction image 225 represents the captured image, the acquisition unit 111 cuts out the determined image area and sets it as the captured image 121. The captured image 121 can be said to be an image obtained by reproducing the captured image captured by the camera unit 210 in the inspection support device 100.

[0036] The diagnosis unit 112 generates a diagnostic image by cutting out an image of a diagnostic region from the captured image 121. Then, the image data of the generated diagnostic image is input to the analytical neural network 151, which calculates the estimated probability that a lesion is present in the image, and passes the result to the display control unit 113. The display control unit 113 develops and arranges the captured image data of the captured image 121 received from the acquisition unit 111 and the diagnostic information including the estimated probability received from the diagnosis unit 112 according to a preset display mode, and displays them on the display monitor 120.

[0037] Specifically, as shown in the lower diagram of Fig. 4, for example, the captured image 121 is arranged to the right, and the diagnostic information 122 is organized into elements such as numerical information indicating estimated probability, pie chart-like graphics, etc. and arranged to the left. Also, a diagnostic region frame 121a is displayed superimposed so that it is possible to visually recognize which region of the captured image 121 has been designated as the diagnostic region. Note that the diagnostic region frame 121a displayed here is preferably displayed in a different manner from the diagnostic region frame 121a in Fig. 3 superimposed on the captured image 121 before diagnosis so that it can be recognized as a diagnosed region.

[0038] As described above, in the examination support device 100 according to this embodiment, the acquisition unit 111 acquires the display signal sent from the endoscope system 200. The diagnosis unit 112 infers the doctor's diagnostic instructions by utilizing changes in the captured image 121 that is successively generated from the display signal.

[0039] 5 is a diagram illustrating a procedure for determining whether a freeze has occurred based on changes in the captured image 121 and determining the diagnostic target image. The examination support device 100 receives a display signal at a cycle of, for example, 60 fps, and the acquisition unit 111 generates the captured image 121 by cutting out an image area from the signal playback image 225 as described above each time. The captured images 121 generated sequentially in this manner can be treated as frame images Fr that may change over time. FIG. 5 shows the sequentially generated frame images Fr1, Fr2, Fr3, Fr4, Fr5, ...

[0040] The camera unit 210 continuously repeats image capture and sequentially transmits image signals as frame images to the endoscope system 200. While the acquisition unit 111 receives a display signal including such image signals from the endoscope system 200, it sequentially generates frame images Fr that change over time and correspond to images captured by the camera unit 210. This period is referred to as a moving image period.

[0041] When the doctor wants to carefully observe the captured image 121 displayed on the display monitor 120, the doctor presses the freeze button provided on the operation unit of the camera unit 210. The endoscope system 200 detects that the freeze button has been pressed, and freezes the image signal captured by the camera unit 210 at that timing. That is, a freeze in which a series of still images is displayed starts from the timing when the freeze button is pressed.

[0042] While the freeze button of the camera unit 210 is pressed, the acquisition unit 111 sequentially generates frame images Fr corresponding to still images. This period is referred to as the still image period. In the example of Fig. 4, the moving image period extends up to frame image Fr2, and the still image period begins with frame image Fr3, which continues for a while thereafter.

[0043] The diagnosis unit 112 can detect that a still image period has started when, for example, the difference between the previous and next frame images falls below a threshold. The diagnosis unit 112 recognizes that a doctor has given a diagnosis instruction when the still image period starts. The diagnosis unit 112 determines that the frame image at the time when the start of the still image period can be detected is the diagnostic target image IM.

[0044] Next, a series of processing steps from when an image area is determined for a signal reproduction image until the end of the inspection will be described. Fig. 6 is a flow chart for explaining the processing steps of the arithmetic processing unit 110.

[0045] In step S101, the acquisition unit 111 acquires a display signal from the endoscope system 200, and in step S102, cuts out a captured image 121 from a signal playback image obtained by expanding the display signal, and hands it over to the display control unit 113. In step S103, the display control unit 113 superimposes a diagnostic area frame 121a on the captured image 121 received from the acquisition unit 111, and displays it on the display monitor 120 as described with reference to FIG.

[0046] In step S104, diagnosis unit 112 compares the previous frame image with the current frame image to determine whether or not the current frame image is a diagnostic target image, as described with reference to Fig. 5. If it is determined that the current frame image is not a diagnostic target image, the process proceeds to step S108, and if it is determined that the current frame image is a diagnostic target image, the process proceeds to step S105.

[0047] In step S105, the diagnosis unit 112 cuts out a diagnostic region from the captured image 121 that has been determined to be an image to be diagnosed. Then, in step S106, the image of the cut out diagnostic region is input to the analytical neural network 151 read out from the storage unit 150, which calculates an estimated probability that a lesion is present in the image, and passes the result to the display control unit 113. In step S107, the display control unit 113 displays the calculation result received from the diagnosis unit 112 on the display monitor 120 together with the captured image 121 received from the acquisition unit 111. When a certain time (e.g., 3 seconds) has elapsed since the start of display, the process proceeds to step S108.

[0048] When the process proceeds to step S108, the calculation processing unit 110 checks whether or not an instruction to end the examination has been received. If an instruction to end the examination has not been received, the process returns to step S101 and the series of processes is repeated. If an instruction to end the examination has been received, the series of processes is terminated.

[0049] Next, several modified examples of this embodiment will be described. FIG. 7 is a diagram illustrating the step-by-step diagnostic process for a divided region in a first modified example. In the above-described embodiment, the diagnosis unit 112 diagnosed the presence or absence of a lesion only in the diagnostic region set in the center. However, there are often cases where the central part of the central region that the doctor is interested in is unlikely to be a lesion, but there are areas around it that are likely to be a lesion. Therefore, in this modified example, if the diagnosis unit 112 does not find a lesion that exceeds a predetermined criterion in the central diagnostic region, it diagnoses the adjacent region adjacent to the central part.

[0050] Specifically, an extended diagnostic region frame 121b is set to surround a diagnostic region frame 121a in the captured image 121. The region surrounded by the diagnostic region frame 121a is set as a first priority region, and the region surrounded by the extended diagnostic region frame 121b but excluding the first priority region is set as a second priority region. The first priority region corresponds to the diagnostic region in the above-described embodiment.

[0051] The diagnosis unit 112 first cuts out the first priority region from the captured image 121 and inputs it to the analytical neural network 151. As a result, if an estimated probability exceeding a reference value (e.g., 50%) is calculated, the display control unit 113 displays the result, for example, as shown in the lower left diagram of FIG. 7. On the display screen, for example, the captured image 121 in which the first priority region is highlighted and the diagnostic information 122, which is the diagnosis result, are displayed side by side so that the first priority region can be visually recognized as the diagnostic region. In this case, the observation target 900 adjusted by the doctor to fit within the diagnostic region frame 121a is included in the highlighted first priority region.

[0052] If the diagnosis result for the first priority region is equal to or less than the reference value, the diagnosis unit 112 subsequently extracts the second priority region from the captured image 121 and inputs it to the analytical neural network 151. If an estimated probability exceeding the reference value is calculated as a result, the display control unit 113 displays the result, for example, as shown in the lower right diagram of FIG. 7. On the display screen, for example, the captured image 121 with the second priority region highlighted is displayed alongside the diagnostic information 122, which is the diagnosis result, so that the second priority region can be visually recognized as the diagnostic region. In this case, the observation target 900 that was not intended by the doctor or that failed to fit within the diagnostic region frame 121a is included in the highlighted second priority region. Note that if the diagnosis result for the second priority region is also equal to or less than the reference value, the display control unit 113 skips displaying the diagnosis result or displays a message indicating that no lesion with a probability exceeding the reference value was found.

[0053] In addition, when a diagnostic instruction is not detected, the display control unit 113 may superimpose and display the extended diagnostic region frame 121b in addition to the diagnostic region frame 121a on the almost real-time captured image 121 generated sequentially from the display signal. In this way, by including the second priority region adjacent to the first priority region as a diagnostic target in addition to the first priority region, it is possible to give the doctor new insights and cover insufficient adjustment of the imaging position. Furthermore, by setting the second priority region adjacent to the first priority region instead of the entire region of the captured image 121, it is possible to prevent erroneous results due to artifacts, etc. from being displayed.

[0054] Next, a second modified example will be described. Fig. 8 is a hardware configuration diagram of a test support device 100' in the second modified example. The hardware configuration of the test support device 100' differs from the hardware configuration of the test support device 100 according to the above embodiment in that it includes a modification unit 114. Since the other configurations are the same, the description thereof will be omitted.

[0055] The change unit 114 changes the setting of a partial region of the captured image 121 that is to be diagnosed by the diagnosis unit 112. In the above-described embodiment, the diagnosis region was set in the center of the captured image 121. However, depending on the organ or observation location to be observed and the structure of the camera unit 210, it may be difficult to guide the observation target 900 to the center of the captured image 121. Therefore, the examination support device 100′ in this modified example is provided with the change unit 114 so that the user, i.e., a doctor, can set the diagnosis region in advance.

[0056] FIG. 9 shows an example of a user interface screen for setting a diagnostic area in the second modified example. The diagnostic area is set before the start of an examination. When a doctor selects "Set diagnostic area" from the menu items, a setting screen opens. The setting screen displays a title 123 for "Set diagnostic area," a selection button 124 for each option, and selection items 125. Options include "Standard," which fixes the diagnostic area to the center, as well as "Select observation area," "Select frame," and "Direct selection," and the doctor can select one of them using the selection button 124. Note that, although a configuration in which four options can be selected is described here, two or more of these may be provided, or other options may be added.

[0057] When the physician selects the observation region selection button 124, an observation region selection window 126 appears. The observation region selection window 126 displays a title 123 of "Observation Region Selection," as well as selection buttons 124 and selection items 125 for each option. The options include organ names for the observation region, such as "Stomach," "Esophagus," and "Duodenum," and a scroll indicator 127 is displayed to indicate that additional selection items exist below. The physician can select one of the options using the selection button 124, and then selects the decision button 128 to set the selected option. Convenient regions for each observation region are pre-assigned to the partial regions serving as diagnostic regions, and the change unit 114 changes the setting of the partial region in response to the physician's selection and decision. In response to this, the display control unit 113 changes the superimposition position of the diagnostic region frame 121a on the captured image 121.

[0058] When the doctor selects the frame selection button 124, a frame selection window 126 appears. The frame selection window 126 displays a "Frame Selection" title 123, selection buttons 124 for each option, and selection items 125 for each option. Options include "Right," "Left," and "Top," which indicate relative positions with respect to the captured image 121, and a scroll indicator 127 is displayed to indicate that other selection items exist below. The doctor can select one of the options using the selection button 124, and then selects the confirm button 128 to set the selected option. The partial areas as the diagnostic area are predetermined according to the relative positions indicated by each item, and the change unit 114 changes the setting of the partial area in response to the doctor's selection and confirmation. In response to this, the display control unit 113 changes the superimposition position of the diagnostic area frame 121a on the captured image 121.

[0059] When the doctor selects the direct selection button 124, a direct selection window 126 appears. The direct selection window 126 displays a "Direct Selection" title 123 and a sample image 121' corresponding to the captured image 121. The doctor directly specifies the area by tracing the desired diagnostic area on the sample image 121' with a fingertip. Specifically, a touch panel overlaid on the display monitor 120 detects the contact position of the fingertip, and a trace line 129 is displayed as a graphic overlaid on the sample image 121' according to the contact trajectory. When the trace line closes, a partial area of ​​the entire captured image 121 is confirmed, and the area specification is completed. However, if the specified diagnostic area is too small or too large, the diagnosis unit 112 may not be able to make an appropriate diagnosis. In such cases, the change unit 114 adjusts the size of the specified diagnostic area so that it fits within a predetermined diagnosable range. In response to this, the display control unit 113 changes the superimposition position of the diagnostic area frame 121a on the captured image 121. When the size of the partial region is changed, the size of the superimposed diagnosis region frame 121a is also changed. Note that the region designation is not limited to the method of detecting the trajectory of the touching fingertip, and other methods may also be used.

[0060] Next, a third modified example will be described. In all of the embodiment, first modified example, and second modified example described so far, the diagnosis unit 112 generates a diagnostic region image by cutting out a predetermined partial region from the captured image 121, and inputs the diagnostic region image to the analytical neural network 151. In other words, a diagnosis is made only on a partial region of the captured image 121. However, in the third modified example, the entire region of the captured image 121 is input to the analytical neural network 151 as a diagnostic image. Figure 10 is a diagram explaining the processing up to displaying the diagnosis result in the third modified example.

[0061] The process from when the acquisition unit 111 acquires the display signal sent from the endoscope system 200 to when the determined image area is cut out to become the captured image 121 is the same as that in the embodiment of Fig. 4. The captured image 121 can be said to be an image in which the image captured by the camera unit 210 is reproduced by the examination support device 100.

[0062] The diagnostic unit 112 inputs the entire captured image 121 as a diagnostic image to the analytical neural network 151, and calculates the estimated probability that a lesion exists in the image. The analytical neural network 151 outputs regions in the image that are suspected of containing a lesion, along with the estimated probability, as diagnostic results. In the example of FIG. 10, the upper left region showing an estimated probability of 60% and the region near the center showing an estimated probability of 85% are output. The diagnostic unit 112 distinguishes the diagnostic results output by the analytical neural network 151 into diagnostic results for a preset diagnostic region frame 121a and diagnostic results for regions other than the diagnostic region frame 121a, and transfers the former to the display control unit 113.

[0063] The display control unit 113 develops and arranges the captured image data of the captured image 121 received from the acquisition unit 111 and the diagnostic information including the estimated probability received from the diagnosis unit 112 according to a preset display mode and displays them on the display monitor 120. As in the embodiment of Fig. 4, the display control unit 113 arranges the captured image 121 to the right, and arranges the diagnostic information 122 as elements such as numerical information indicating the estimated probability for the diagnostic area frame 121a, pie chart-like graphics, etc. to the left. In addition, the diagnostic area frame 121a is displayed superimposed so that it is possible to visually recognize which area of ​​the captured image 121 is set as the diagnostic area.

[0064] The diagnosis unit 112 may distinguish between the diagnosis result for the diagnostic region frame 121a set in advance and the diagnosis result for the region other than the diagnostic region frame 121a, and transfer both of them to the display control unit 113. In this case, the display control unit 113 may use the diagnosis result for the diagnostic region frame 121a set in advance to generate the display of the diagnostic information 122. Furthermore, the diagnosis unit 112 may store the diagnosis result for the diagnostic region frame 121a and the diagnosis result for the region other than the diagnostic region frame 121a in the storage unit 150 together with the captured image 121, distinguishing between them, so that a doctor can check them later.

[0065] Fig. 11 is a flowchart illustrating the processing procedure of the arithmetic processing unit in the third modified example. The same steps as those in the flowchart of Fig. 6 are denoted by the same step numbers, and their description will be omitted.

[0066] If the diagnosis unit 112 determines in step S104 that the current frame image is a diagnostic target image, the process proceeds to step S201. In step S201, the diagnosis unit 112 inputs the entire captured image 121, which has been determined to be a diagnostic target image, to the analytical neural network 151 as a diagnostic image, and causes the analytical neural network 151 to calculate an estimated probability that a lesion exists in the image. The diagnosis unit 112 proceeds to step S202, where it extracts a diagnostic result for a preset diagnostic region frame 121a from the diagnostic results output by the analytical neural network 151, and transfers the extracted diagnostic result to the display control unit 113.

[0067] When the process proceeds from step S202 to step S107, the display control unit 113 displays the diagnosis result received from the diagnosing unit 112 on the display monitor 120 together with the captured image 121 received from the acquiring unit 111.

[0068] In the third modified example, a doctor can also recognize the estimated probability that a target area of ​​interest is a lesion by moving the target area into the diagnostic area frame. The first and second modified examples can also be combined with the third modified example. In the first modified example, when the diagnostic result of the set diagnostic area falls below a preset criterion, the adjacent area adjacent to the diagnostic area is diagnosed. In the third modified example, the diagnostic unit 112 does not perform the diagnosis stepwise, but outputs the diagnostic result for the diagnostic area frame 121a separately from the diagnostic result for the area other than the diagnostic area frame 121a. Therefore, when combining the first modified example with the third modified example, the display control unit 113, when displaying the diagnostic result of the second priority area, may extract and display the diagnostic result corresponding to the second priority area from the diagnostic results for the area other than the diagnostic area frame 121a output by the diagnostic unit 112. However, the diagnostic result corresponding to the second priority area is displayed only when an estimated probability exceeding a reference value (e.g., 50%) is calculated, and is not displayed when the estimated probability is below the reference value.

[0069] When the second modified example is combined with the third modified example, the diagnosis unit 112 distinguishes between diagnostic results depending on the position of the diagnostic region frame 121a changed by the change unit 114. For example, when "right" is selected in the frame selection, the diagnosis unit 112 distinguishes between diagnostic results for the diagnostic region frame 121a set to the right of the center and diagnostic results for regions other than the diagnostic region frame 121a, and outputs the distinguished results. The first modified example and the second modified example may also be combined with the third modified example.

[0070] Although the above describes each of the modified examples, the modified examples are not limited to these. For example, the setting of the diagnostic area does not need to be performed prior to the start of the examination, but may be accepted during the examination. If the doctor can directly specify the diagnostic area on the real-time captured image 121, it is possible to appropriately deal with artifacts that appear. Furthermore, the change unit 114 may estimate the current observation area by inputting the real-time captured image 121 into a neural network for site diagnosis, and change the setting to a partial area appropriate for the estimated observation area. In this case, it is sufficient that the partial areas appropriate for each recognizable observation area are associated in advance as described above. The display control unit 113 changes the superimposition position of the diagnostic area frame 121a on the captured image 121 in accordance with the processing of the change unit 114. Note that when the partial area is automatically and sequentially changed according to the observation area in this manner, it is preferable to set a certain interval (e.g., one minute) for determining whether or not to change the partial area.

[0071] Furthermore, in the present embodiment described above, the diagnosis unit 112 performs diagnostic processing on the diagnosis area every time a diagnostic instruction is detected, and the display control unit 113 displays the results on the display monitor 120. However, the diagnostic processing and result display may be performed all at once after the doctor has finished operating the camera unit 210. If the diagnostic area frame 121a is displayed superimposed on at least the captured image 121 in almost real time, the doctor can press the release button to appropriately record the diagnostic target image in the endoscope system 200, and can know the diagnosis results from the diagnosis unit 112 even after the fact.

[0072] Furthermore, in the above-described embodiment, it is assumed that the endoscopic system 200 and the examination support device 100 are connected via a connection cable 250, but a wireless connection may also be used instead of a wired connection. Furthermore, while an embodiment has been described in which the endoscopic system 200 outputs a display signal to the outside and the examination support device 100 uses this display signal, the format of the output signal does not matter as long as the image signal provided by the endoscopic system 200 to an external device includes an image signal of an image captured by the camera unit 210. Furthermore, in the above-described embodiment, it is assumed that the camera unit 210 included in the endoscopic system 200 is a flexible endoscope. However, even if the camera unit 210 is a rigid endoscope, there is no difference in the configuration or processing procedure of the examination support device 100. Note that the diagnosis by the diagnosis unit in this embodiment is merely to assist the doctor's diagnosis, and the final decision is made by the doctor. [Explanation of symbols]

[0073] 100, 100'... examination support device, 110... arithmetic processing unit, 111... acquisition unit, 112... diagnosis unit, 113... display control unit, 114... change unit, 120... display monitor, 121... captured image, 121'... sample image, 121a... diagnosis area frame, 121b... extended diagnosis area frame, 122... diagnosis information, 123... title, 124... selection button, 125... selection item, 126... window, 127... scroll indicator, 128... decision button, 129... trace line, 130... input / output interface, 140... input device, 150... memory unit, 151... analytical neural network, 200... endoscope system, 210... camera unit, 220... system monitor, 221... captured image, 222... examination information, 225... signal playback image, 250... connection cable, 900... observation target, 910... artifact

Claims

1. an acquisition unit that sequentially acquires images captured by a camera unit inserted into the body of a subject; a diagnosis unit that performs a diagnosis of the inside of the body based on a set partial region of the captured image; a display control unit that sequentially displays the captured images on a display unit in a manner that allows a user to recognize the partial area as a display area that is larger than the partial area; Equipped with the partial region is a fixed region that is set in advance with respect to the entire captured image, the display control unit displays the partial area in a manner that the partial area can be recognized before receiving a diagnostic instruction from a user, The diagnostic unit performs the diagnosis in response to the diagnostic instruction from a user. Inspection support equipment.

2. The inspection support device according to claim 1 , further comprising a change unit for changing the setting of the partial region.

3. The examination support device according to claim 2 , wherein the change unit changes the partial area to one selected by a user from a plurality of partial areas prepared in advance.

4. 4. The examination support device according to claim 2, wherein the change unit changes the setting of the partial region in accordance with one selected by a user from a plurality of observation regions prepared in advance.

5. 5. The examination support device according to claim 2, wherein when a partial area of ​​the display area is designated by a user, the change unit adjusts the partial area to an area that can be diagnosed by the diagnosis unit and sets the partial area as the partial area.

6. The examination support device according to claim 2 , wherein the change unit changes the setting of the partial region depending on which observation site is imaged in the captured image acquired by the acquisition unit.

7. 7. The inspection support device according to claim 1, wherein when the diagnosis unit diagnoses the partial region, the display control unit displays the captured image on the display unit for a certain period of time together with the diagnosis result in a manner that allows the partial region to be recognized.

8. the diagnosing unit, when not finding a lesion exceeding a predetermined criterion in the set partial region, diagnoses an adjacent region adjacent to the set partial region; The inspection support device according to claim 7, wherein when the diagnosis unit diagnoses the adjacent area, the display control unit freezes the captured image for a certain period of time and displays it on the display unit in a manner that allows the adjacent area to be recognized along with the diagnosis result.

9. An examination support method using an examination support device including an acquisition unit, a display control unit, and a diagnosis unit, an acquiring step in which the acquiring unit sequentially acquires images captured by a camera unit inserted into the body of the subject; a display step in which the display control unit sequentially displays the captured image on the display unit as a display area that is larger than the set partial area of ​​the captured image in a manner that allows a user to recognize the set partial area of ​​the captured image; a diagnosis step in which the diagnosis unit performs a diagnosis of the inside of the body based on the partial region of the captured image; and the partial region is a fixed region that is set in advance with respect to the entire captured image, the display step displays the partial area in a manner that the partial area can be recognized before receiving a diagnostic instruction from a user, The diagnosis step performs the diagnosis in response to a diagnosis instruction from a user. Inspection support methods.

10. An examination assistance program for controlling an examination assistance device including an acquisition unit, a display control unit, and a diagnosis unit, an acquiring step of causing the acquiring unit to sequentially acquire images captured by a camera unit inserted into the body of the subject; a display step of causing the display control unit to sequentially display the captured image on the display unit as a display area that is larger than the set partial area of ​​the captured image in a manner that allows a user to recognize the set partial area of ​​the captured image; a diagnosis step of causing the diagnosis unit to diagnose the inside of the body based on the partial region of the captured image; on the computer, the partial region is a fixed region that is set in advance with respect to the entire captured image, the display step displays the partial area in a manner that the partial area can be recognized before receiving a diagnostic instruction from a user, The diagnosis step performs the diagnosis in response to a diagnosis instruction from a user. Inspection support program.

Images