ENDOSCOPE PROCESSOR, INFORMATION PROCESSING METHOD, AND COMPUTER PROGRAM

Abstract

This endoscope processor comprises: a determination unit that determines whether or not a start condition for starting the acquisition of a moving image having a resolution equal to or higher than a still image photographed by an endoscope and having a predetermined limited time length is satisfied; and a moving image acquisition unit that starts the acquisition of the moving image when the determination unit determines that the start condition is satisfied.

Description

[Technical field]

[0001] The present technology relates to an endoscope processor, an information processing method, and a computer program. [Background technology]

[0002] Electronic endoscopes, which include an electronic endoscope with an objective optical system and an image sensor built in the distal end of the insertion tube of the endoscope, and an electronic endoscope processor that processes the image signal output from the electronic endoscope to generate a video signal that can be displayed on a monitor, are widely used for the diagnosis of subjects. The electronic endoscope processor performs various signal processing such as image processing for distortion correction and scratch correction, gamma correction, white balance adjustment, and enhancement processing on the captured image to provide a high-quality image suitable for a doctor's diagnosis. The doctor makes a diagnosis based on these images.

[0003] In addition, electronic endoscope devices also perform many other processes, such as control of the objective optical system and the image sensor, and reception of various operation requests, and it is required that these processes be performed without delay.Patent Document 1 discloses an electronic endoscope system and the like that can prevent an increase in the load on the processor's operating system, operational delays, and data loss without using an expensive CPU. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2018-7928 A Summary of the Invention [Problem to be solved by the invention]

[0005] In endoscopic examinations, there are cases where the doctor performs a diagnosis during the endoscopic procedure, and cases where the diagnosis is performed at a later date based on the recorded images. When a diagnosis is performed at a later date, the diagnosis is usually based on a single saved still image, so the doctor performing the diagnosis is required to have high skills and experience. However, there is a problem in that recording a video from the start to the end of the examination with the same image quality as the still images during the procedure increases the capacity.

[0006] An object of the present disclosure is to provide an endoscope processor, an information processing method, and a computer program for acquiring moving images captured by an endoscope over a limited, predetermined time period. [Means for solving the problem]

[0007] An endoscopic processor according to one aspect of the present disclosure includes a determination unit that determines whether a start condition is satisfied for starting acquisition of a moving image limited to a predetermined length of time and having a resolution equal to or greater than that of a still image captured by an endoscope, and a moving image acquisition unit that starts acquiring the moving image when the determination unit determines that the start condition is satisfied. Effect of the Invention

[0008] According to the present disclosure, it is possible to obtain a moving image captured by an endoscope for a predetermined time period. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic diagram illustrating an overall configuration of an endoscope system according to a first embodiment. [Diagram 2] 2 is a block diagram showing a configuration example of an endoscope processor; [Diagram 3] FIG. 4 is a diagram showing an example of information stored in an image DB; [Figure 4] FIG. 2 is a functional block diagram of an endoscope processor. [Diagram 5] FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. [Figure 6]5 is a flowchart showing an example of a processing procedure executed by an endoscope processor. [Figure 7] FIG. 11 is a functional block diagram of an endoscope processor according to a second embodiment. [Figure 8] 10 is a flowchart showing an example of a connection process procedure executed by the endoscope processor. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will now be described in detail with reference to the drawings showing embodiments thereof.

[0011] (First embodiment) 1 is a schematic diagram for explaining the overall configuration of an endoscope system in a first embodiment. The endoscope system in the first embodiment includes an endoscope 1, an endoscope processor 2, and a display device 3. The endoscope 1 is inserted inside a subject, and outputs image data of an image captured inside the subject to the endoscope processor 2. The endoscope processor 2 performs image processing on the image data input from the endoscope 1, and outputs the image data after the image processing to the display device 3. The display device 3 displays an image based on the image data input from the endoscope processor 2. Each device transmits and receives electrical signals, video signals, etc. via a connector.

[0012] The endoscope 1 is, for example, an upper digestive tract endoscope or a colonoscope. The endoscope 1 includes an insertion section 11 that is inserted into a tubular organ of a subject, and a scope connector 14 that is connected to the insertion section 11 via an operation section 12 and a universal cord 13, and is used by connecting to an endoscope processor 2 via the scope connector 14. An imaging device 15 is built into the tip of the insertion section 11.

[0013] The insertion section 11 has a flexible tube 111 connected to the operation section 12 and formed relatively long, and a bending section 112 connected coaxially to the flexible tube 111 and formed relatively short so as to be freely bent. The bending mechanism in the bending section 112 is a well-known mechanism incorporated in a typical endoscope, and is configured so that the bending section 112 is bent by pulling an operation wire linked to the operation of the operation section 12. The direction of the tip portion changes according to the bending movement caused by the above operation, and the imaging area of ​​the imaging device 15 moves.

[0014] The imaging device 15 includes an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an objective optical system for forming an image. A driver circuit for driving the imaging device 15 is disposed at a connection portion (inside the scope connector 14) with the endoscope processor 2, and includes a CPU (Central Processing Unit), a timing generator (TG), an analog signal processing circuit (AEF), and the like. The driver circuit takes in the RGB color signals output from the solid-state imaging element in response to a clock signal output from the TG, and outputs the digital image data obtained by performing necessary processing such as noise removal, amplification, and AD conversion to the endoscope processor 2.

[0015] The operation unit 12 is provided to be held by an operator to perform various operations, and is provided with a control button 121 for recording images including still images and high-quality moving images. When the control button 121 is pressed, the endoscope 1 captures a still image and a high-quality moving image of a predetermined length of time. In addition to the control button 121, the operation unit 12 may be provided with other operation buttons for accepting various operations by the operator. The operation unit 12 outputs an operation signal indicating an operation performed on the operation unit 12 to the endoscope processor 2.

[0016] In another embodiment, the operation unit 12 may be configured to accept a still image shooting operation and a high-quality video shooting operation independently. For example, the control button 121 is configured to be able to detect two-stage pressing operations, namely, a still image shooting operation and a high-quality video shooting operation. The control button 121 accepts a still image shooting operation by pressing it in one stage, and accepts a high-quality video shooting operation by pressing it down another stage. Note that a plurality of control buttons 121 may be provided, and a configuration may be adopted in which a still image shooting operation and a high-quality video shooting operation are accepted by different control buttons 121.

[0017] The endoscope processor 2 is an information processing device for processing image data captured by the imaging device 15 of the endoscope 1 and transmitted via the universal code 13. The image data processed by the endoscope processor 2 is output to the display device 3.

[0018] The display device 3 is, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device, and displays image data output from the endoscope processor 2 and the like.

[0019] 2 is a block diagram showing an example of the configuration of the endoscope processor 2. The endoscope processor 2 includes a control unit 21, a storage unit 22, an input unit 23, an output unit 24, a light source 25, and the like.

[0020] The control unit 21 has a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The CPU of the control unit 21 causes the endoscope processor 2 to function as the endoscope processor 2 in this embodiment by expanding a program 2P stored in the ROM or the storage unit 22 onto the RAM and executing it.

[0021] The control unit 21 is not limited to the above configuration. The control unit 21 may be one or more arithmetic circuits or control circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, a volatile or non-volatile memory, etc. The control unit 21 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given to when an instruction to end measurement is given, and a counter that counts numbers.

[0022] The storage unit 22 is configured by a non-volatile memory such as an EPROM (Erasable Programmable Read Only Memory) or a recording device equipped with a hard disk. The storage unit 22 stores the program 2P, as well as other programs and data referenced by the control unit 21. The storage unit 22 may be configured by a plurality of storage devices, or may be an external storage device connected to the endoscope processor 2. In another embodiment, the storage unit 22 may be a portable recording medium such as a USB memory or an SD card, and may be detachable from the endoscope processor 2.

[0023] The program 2P may be provided to the endoscope processor 2 using a recording medium 2A on which the program 2P is readably recorded. The recording medium 2A is, for example, a portable memory such as a CD-ROM, a USB memory, an SD card, a micro SD card, or a Compact Flash (registered trademark). In this case, the control unit 21 may read the program 2P from the recording medium 2A via a reading unit (not shown) and install the read program 2P in the storage unit 22. Furthermore, when the endoscope processor 2 includes a communication unit that communicates with an external device, the program 2P may be provided to the endoscope processor 2 by communication via the communication unit. In this case, the control unit 21 may acquire the program 2P through the communication unit and install the acquired program 2P in the storage unit 22.

[0024] The storage unit 22 further stores a learning model 2M and an image DB 222 (Data Base: database). The image DB 222 is a database that registers image information related to images captured by an endoscope. The learning model 2M is a classifier that identifies lesions in images captured by an endoscope, and is a learning model generated by machine learning. The learning model 2M is defined by its definition information. The definition information of the learning model 2M includes, for example, structural information and layer information of the learning model 2M, node information included in each layer, and learned parameters. The storage unit 22 stores definition information related to the learning model 2M.

[0025] The input unit 23 includes a connection interface for connecting input devices such as a touch panel, a keyboard, various switches, etc. The input unit 23 inputs to the control unit 21 an input signal generated in response to an external operation on these input devices.

[0026] The output unit 24 includes a connection interface for connecting the display device 3. The output interface of the output unit 24 may be an output interface for outputting an analog video signal, or an output interface for outputting a digital video signal such as DVI (Digital Visual Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). Under the control of the control unit 21, the output unit 24 outputs an image signal for display to the display device 3 to display an image.

[0027] The light source 25 includes a light source that emits illumination light used to illuminate an object to be observed. The light source 25 is, for example, a semiconductor light source such as a light emitting diode (LED) having multiple colors with different wavelength ranges, a combination of a laser diode and a phosphor, or a xenon lamp. The control unit 21 controls the turning on and off of the light source 25 and the change in brightness. In this embodiment, the endoscope processor 2 is an integrated light source type, but is not limited to this. For example, the endoscope processor 2 may be a light source separated type that is separated from the light source device.

[0028] Furthermore, the endoscope processor 2 may include a communication unit. The endoscope processor 2 may transmit image data and the like to an external device and store this information in the external device.

[0029] In this embodiment, the endoscope processor 2 will be described as being one information processing device, but the processing may be distributed among a plurality of devices, or may be configured as a virtual machine.

[0030] 3 is a diagram showing an example of the contents of information stored in the image DB 222. The control unit 21 generates still images and videos based on captured image data acquired from the endoscope 1 and stores them in the image DB 222. The image DB 222 stores images, time information, subject IDs, start conditions, etc. in association with image IDs for identifying image files.

[0031] The image may include an image acquired during the examination of the endoscope 1. In this embodiment, the image includes a still image and a moving image, and the moving image further includes a low-quality moving image and a high-quality moving image. That is, the moving image includes moving images of different image quality of the same imaging target at the same time captured by the same imaging device. The still image is an image acquired and recorded according to an instruction from an operator. The low-quality moving image is an image captured from the start to the end of the examination of the endoscope 1 and recorded at a low resolution. The high-quality moving image is an image acquired according to a predetermined starting condition and recorded at a high resolution. In addition to the above, the image generated based on the captured image data acquired from the endoscope 1 includes an endoscopic image acquired from the start to the end of the examination of the endoscope 1 and displayed on the display device 3. The time information includes information indicating the recording time of each image, and for example, a time stamp is stored. The subject ID includes information identifying the subject that is the imaging target of each captured image data, and for example, a subject ID (patient ID) is stored. The starting condition includes information indicating the starting condition that led to the start of the imaging of the high-quality moving image, such as acceptance of a start operation, detection of a lesion, and a movement amount being equal to or less than a predetermined value. In the example of Fig. 3, the start conditions are normalized and registered using type codes from 1 to 3. Details of the start conditions will be described later. Note that Fig. 3 is an example, and the stored contents of the image DB 222 are not limited.

[0032] In normal endoscopic examinations, due to the storage capacity of the endoscope processor 2, it is common to store high-resolution still images taken at a specified examination location and low-resolution video taken throughout the entire procedure. When a doctor makes a diagnosis at a later date, the diagnosis is made based on the stored still images. Diagnosis from a single still image requires skilled techniques. Therefore, this endoscope system further stores high-resolution video limited to a specified length of time, making it possible to provide a lot of information from still images and videos recorded in high quality even when a diagnosis is made based on the stored data.

[0033] 4 is a functional block diagram of the endoscope processor 2. The control unit 21 of the endoscope processor 2 is configured to execute a program 2P stored in the storage unit 22 to function as an image processing unit 211, a reception unit 212, a lesion detection unit 213, a movement amount detection unit 214, a determination unit 215, an identification information acquisition unit 216, a still image acquisition unit 217, a low-quality video acquisition unit 218, and a high-quality video acquisition unit 219.

[0034] Image information for one frame that has been subjected to various processes such as noise removal is input as captured image data from the driver circuit of the endoscope 1 to the image processing unit 211. When an endoscopic examination is started, the image processing unit 211 outputs the captured image data to the still image acquisition unit 217, the low-quality video acquisition unit 218, the lesion detection unit 213, and the movement amount detection unit 214 in accordance with a control signal output at a predetermined interval from the start to the end. The image processing unit 211 also outputs the captured image data to the still image acquisition unit 217 and the high-quality video acquisition unit 219 at a timing instructed by the determination unit 215.

[0035] The reception unit 212 receives a start operation by the operator of the endoscope 1 to instruct the start of acquisition of high-quality moving images. The start operation to instruct the start of acquisition of high-quality moving images is, for example, pressing the control button 121. In this embodiment, the endoscope processor 2 acquires high-quality moving images simultaneously with still images by pressing the control button 121. When the control button 121 of the endoscope 1 is pressed, an operation signal indicating this is input to the reception unit 212. The reception unit 212 outputs information indicating the acceptance of the start operation to the determination unit 215. The endoscope processor 2 may include a voice input unit such as a microphone (not shown), and the reception unit 212 may receive the start operation by voice via the voice input unit. The reception unit 212 may receive the start operation by operating a touch panel, a keyboard, or the like via the input unit 23.

[0036] Lesion detection unit 213 outputs a lesion in the captured image data based on the captured image data output from image processing unit 211. Lesion detection unit 213 includes a function as learning model 2M that has been trained to output a lesion in the image data when specified image data is input. Learning model 2M will be described later. Lesion detection unit 213 outputs information indicating the lesion detection result to determination unit 215. Note that lesion detection unit 213 may detect a lesion using image data obtained by performing various image processing on the captured image data. Lesion detection unit 213 may acquire still image data output from still image acquisition unit 217 described later.

[0037] The movement amount detection unit 214 detects the amount of movement of the endoscope 1 based on the captured image data output from the image processing unit 211. Specifically, the movement amount detection unit 214 detects the amount of movement of the tip of the insertion unit 11 of the endoscope 1 based on the amount of movement of the imaging area included in the captured image data. The movement amount detection unit 214 obtains a frame of captured image data at the detection time and a frame immediately before the detection time, and calculates the amount of movement in the same area included in the two obtained frames, thereby detecting the amount of movement of the endoscope 1. The movement amount detection unit 214 outputs information indicating the movement amount detection result to the determination unit 215. Note that the movement amount detection unit 214 may detect the amount of movement using image data obtained by performing various image processing on the captured image data. The movement amount detection unit 214 may obtain still image data output from a still image acquisition unit 217 described later.

[0038] The method of detecting the amount of movement is not limited to the above example. The movement amount detection unit 214 may detect the amount of movement of the endoscope 1 based on the captured image using other machine learning techniques. The movement amount detection unit 214 may detect the amount of movement of the endoscope 1 based on a detection value acquired using a detection sensor such as an acceleration sensor or a magnetic sensor provided in the endoscope 1. The movement amount detection unit 214 may also detect a change in the amount of operation instead of the amount of movement of the endoscope 1. For example, the movement amount detection unit 214 may acquire operation data related to the operation of the endoscope 1, such as an operation signal of an operation button provided in the operation unit 12, and detect a history of the amount of operation of the endoscope 1 based on a log of the acquired operation data.

[0039] The determination unit 215 determines whether or not it is the timing to record still images and high quality videos, and outputs the determination result to the image processing unit 211. The timing to store the still image data is the timing immediately after the control button 121 is pressed by the operator of the endoscope 1. When the control button 121 of the endoscope 1 is pressed, an operation signal indicating that is input to the determination unit 215. When an operation signal indicating that the control button 121 has been pressed is input, the determination unit 215 gives the image processing unit 211 a control signal for outputting captured image data to the still image acquisition unit 217. The determination unit 215 may acquire information indicating the reception of an operation signal (start operation) via the reception unit 212. The timing for storing the high-quality video data is the timing immediately after a predetermined start condition is satisfied. The start condition is a condition for starting to capture a high-quality video, and includes cases where a diagnosis is likely to be made based on an image captured at a timing when the condition is satisfied. When the determination unit 215 determines that the start condition is satisfied, it provides the image processing unit 211 with a control signal to start outputting the captured image data to the high-quality video acquisition unit 219.

[0040] A first example of the start condition is acceptance of a start operation such as a button operation by an operator to instruct the start of video capture. The determination unit 215 determines whether or not the start operation has been accepted based on information indicating the acceptance of the start operation output from the acceptance unit 212. When the determination unit 215 acquires information indicating the acceptance of the start operation, it determines that the start condition is satisfied and issues an instruction to start acquiring high-quality video data.

[0041] A second example of the start condition is detection of a lesion in an endoscopic image. Determination unit 215 determines whether or not a lesion has been detected based on information indicating the lesion detection result output from lesion detection unit 213. When determination unit 215 obtains a lesion detection result indicating that a lesion has been detected, it determines that the start condition is satisfied and instructs to start obtaining high-quality video data.

[0042] A third example of the start condition is detection that the amount of movement of the endoscope 1 is equal to or less than a predetermined value. The determination unit 215 determines whether or not the amount of movement of the endoscope 1 is equal to or less than a predetermined value based on information indicating the amount of movement detection result output from the movement amount detection unit 214. That is, the determination unit 215 determines whether or not the insertion, rotation operation, etc. of the endoscope 1 has stopped because the operator continues to observe the same location. If the endoscope 1 remains in the same location, there is a high possibility that the subject to be photographed at that location includes a lesion, etc. If the determination unit 215 determines that the amount of movement of the endoscope 1 is equal to or less than the predetermined value, it determines that the start condition is satisfied and instructs to start acquiring high-quality video data.

[0043] As described above, the start condition includes a plurality of conditions. When the determination unit 215 determines that any one of the start conditions is satisfied, it provides the image processing unit 211 with a control signal for starting output of captured image data.

[0044] The identification information acquiring unit 216 acquires subject identification information such as a subject ID. The identification information acquiring unit 216 acquires the subject identification information via the input unit 23 by accepting an input operation of, for example, a keyboard, a touch panel, etc. The identification information acquiring unit 216 outputs the acquired subject identification information to the still image acquiring unit 217, the low-quality moving image acquiring unit 218, and the high-quality moving image acquiring unit 219.

[0045] The still image acquisition unit (still image acquisition section) 217 ​​acquires captured image data output from the image processing section 211 at predetermined intervals from the start to the end of the examination. The still image acquisition unit 217 performs various image processing such as gamma correction, white balance correction, and shading correction on the acquired captured image data to generate an endoscopic image that is easy for the operator to view. The still image acquisition unit 217 sequentially outputs the generated endoscopic images to the display device 3 via the output section 24. An image of the subject is displayed in real time on the display screen of the display device 3, and the operator makes an observation based on the endoscopic image. Furthermore, the still image acquisition unit 217 acquires captured image data output from the image processing unit 211 at a timing according to an instruction from the determination unit 215. The still image acquisition unit 217 acquires the subject identification information output from the identification information acquisition unit 216. The still image acquisition unit 217 generates a still image by performing a predetermined image processing on the acquired captured image data, and acquires it as still image data. Note that the still image data may be endoscopic image data acquired at the same time. The still image data is image data having a high resolution, such as 1980×1080 (full HD). The still image acquisition unit 217 stores the acquired still image data in the storage unit 22 as a still image file in a predetermined file format, such as JPEG or TIFF. In this case, the still image acquisition unit 217 associates the still image data with the subject identification information and stores it in the storage unit 22.

[0046] The low-quality moving image acquiring unit (second moving image acquiring unit) 218 ​​acquires captured image data output from the image processing unit 211 at a predetermined interval from the start to the end of the examination. The low-quality moving image acquiring unit 218 acquires the subject identification information output from the identification information acquiring unit 216. The low-quality moving image acquiring unit 218 performs a predetermined image processing including changing the resolution on the acquired captured image data, generates a low-quality moving image compressed by a predetermined moving image compression method, and acquires it as low-quality moving image data. The low-quality moving image data has a lower resolution than the still image data and the high-quality moving image data. The low-quality moving image acquiring unit 218 stores the acquired low-quality moving image data in the image DB 222 as a low-quality moving image file in a predetermined file format such as MP4 or MXF. In this case, the low-quality moving image acquiring unit 218 associates the low-quality moving image data with the subject identification information and stores it in the storage unit 22.

[0047] The high-quality video acquisition unit (video acquisition unit) 219 acquires captured image data output from the image processing unit 211 at a timing according to an instruction from the determination unit 215. The high-quality video acquisition unit 219 acquires the subject identification information output from the identification information acquisition unit 216. The high-quality video acquisition unit 219 acquires captured image data for a predetermined time from the timing of starting acquisition. The time for the high-quality video acquisition unit 219 to acquire captured image data is, for example, 1 to 7 seconds, and preferably 5 to 7 seconds. When the acquisition time of the high-quality video is 1 to 7 seconds, for example, even if the frame rate is 30 fsp, the image data can be stored with a storage capacity of tens to hundreds of megabytes. Therefore, it is possible to improve the usefulness of information provided to the operator while suppressing excessive pressure on the storage capacity. The endoscope processor 2 acquires the acquisition time of the high-quality video data by accepting a setting input of the shooting time of the high-quality video in advance by the operator using, for example, a setting screen displayed on a touch panel. The time length for acquiring the high-quality video may be determined based on a lower limit or an upper limit of the capacity of the high-quality video data. That is, the high-quality video acquisition unit 219 may start acquiring a specific amount of high-quality video data. For example, the endoscope processor 2 may derive the resolution and time length of the high-quality video according to a preset acquisition capacity, and acquire video data of the derived time length. In acquiring high-quality video data according to a predetermined capacity, the endoscope processor 2 may acquire image data at a frame rate lower than that of low-quality video data, for example. By lowering the frame rate, it is possible to increase the acquisition time of the high-quality video while suppressing the data capacity.

[0048] The high-quality video acquisition unit 219 performs predetermined image processing including changing the resolution on the acquired captured image data for a predetermined time, generates a high-quality video compressed by a predetermined video compression method, and acquires it as high-quality video data. The high-quality video data is image data having a resolution equal to or higher than that of still image data, such as 1980×1080 (full HD). The high-quality video acquisition unit 219 stores the acquired high-quality video data in the image DB 222 as a high-quality video file in a predetermined file format, such as MP4 or MXF. In this case, the high-quality video acquisition unit 219 associates the high-quality video data with subject identification information and stores it in the storage unit 22.

[0049] In another embodiment, the high-quality video acquisition unit 219 may acquire captured image data output from the image processing unit 211 at a predetermined interval from the start to the end of the examination, and generate a high-quality video file limited to a predetermined time in response to an instruction from the determination unit 215. The high-quality video acquisition unit 219 may acquire high-quality video data based on captured image data for a predetermined time period starting from the timing at which the instruction from the determination unit 215 is acquired. Alternatively, the high-quality video acquisition unit 219 may acquire high-quality video data based on captured image data for a predetermined time period acquired going back a predetermined time from the timing at which the instruction from the determination unit 215 is acquired.

[0050] When the image files are output to and stored in an electronic medical record system or the like, it is desirable to store the image files in a format that corresponds to the electronic medical record system. For example, the image files may be stored in the DICOM (Digital Imaging and Communications in Medicine) standard.

[0051] Here, the learning model 2M will be described in detail. Fig. 5 is an explanatory diagram for explaining the configuration of the learning model 2M. The learning model 2M is generated and learned in advance by deep learning using a neural network in the endoscope processor 2 or an external device. The learning algorithm is, for example, CNN (Convolution Neural Network).

[0052] In the example shown in Fig. 5, the learning model 2M includes an input layer for inputting image data, an output layer for outputting the result of identifying the presence or absence of a lesion, and an intermediate layer (hidden layer) for extracting features. The intermediate layer has a plurality of nodes for extracting features of the input data, and passes the features extracted using various parameters to the output layer. When image data is input to the input layer, a calculation is performed in the intermediate layer using the learned parameters, and output information indicating the presence or absence of a lesion is output from the output layer.

[0053] The output layer of the learning model 2M outputs output information that is a classification result of whether or not a lesion is present. The output layer includes nodes that correspond to the set classification results of presence of lesion and absence of lesion, and outputs the accuracy of each of the presence of lesion and absence of lesion as a score. The control unit 21 can set a classification result with a high score or a classification result with a score equal to or greater than a threshold as the output value of the output layer. Note that the output layer may have one output node that outputs the most accurate classification result, instead of having multiple output nodes that output the accuracy of each classification result.

[0054] The control unit 21 performs learning using data that has been collected from a large amount of image data and diagnosis results from endoscopic examinations performed in the past. The control unit 21 learns the learning model 2M by collecting in advance, as training data, a group of information in which information indicating the presence or absence of a lesion, which is a known diagnosis result, is added to input information obtained from endoscopic examinations performed in the past. The control unit 21 learns various parameters and weights that constitute the learning model 2M, for example, using the backpropagation method, so as to output output information indicating the presence or absence of a lesion according to the endoscopic image data.

[0055] Although an example in which the learning model 2M is a CNN has been described above, the learning model 2M is not limited to a CNN. When time-series data is acquired, a neural network other than a CNN, such as a recurrent neural network (RNN) or a long short-term memory (LSTM) network, may be used. The learning model 2M may be a segmentation network that extracts the region of a lesion. It may also be a model trained by other algorithms, such as a support vector machine or a regression tree that does not use a neural network.

[0056] In the present embodiment, an example has been described in which lesion detection unit 213 detects the presence or absence of a lesion in an endoscopic image using learning model 2M, but the method of detecting a lesion is not limited thereto. Lesion detection unit 213 may determine whether or not a lesion is included in an endoscopic image based on the feature amount of the endoscopic image and the feature amount of a lesion portion stored in advance, using a method such as pattern matching or edge detection.

[0057] Fig. 6 is a flowchart showing an example of a processing procedure executed by the endoscope processor 2. The control unit 21 of the endoscope processor 2 executes the processing shown in Fig. 6 in parallel with processing such as generation of an endoscopic image and generation of a low-quality moving image. The control unit 21 executes the following processing, for example, based on the input contents from the input unit 23 connected to the device itself.

[0058] The control unit 21 receives an input operation from, for example, a keyboard, a touch panel, or the like, and thereby acquires subject identification information via the input unit 23 (step S11).

[0059] The control unit 21 acquires captured image data from the endoscope 1 (step S12). Next, the control unit 21 determines whether or not a start condition is satisfied. The control unit 21 determines whether or not a start operation has been accepted as a first start condition (step S13). The control unit 21 acquires an operation signal by pressing the control button 121 via the acceptance unit 212, and executes a process of outputting information indicating the acceptance of the start operation. If the information indicating the acceptance of the start operation is acquired and it is determined that the start operation has been accepted (step S13: YES), the control unit 21 determines that the first start condition is satisfied, and proceeds to the process of starting the acquisition of high-quality video data in step S16. If information indicating acceptance of a start operation has not been acquired and it is determined that a start operation has not been accepted (step S13: NO), the control unit 21 determines that the first start condition is not satisfied and proceeds to the determination of other start conditions in steps S14 and S15.

[0060] The control unit 21 determines whether or not a lesion is detected from the captured image as the second start condition. (Step S14). Control unit 21 executes a process of outputting information indicating the lesion detection result by lesion detection unit 213. Specifically, control unit 21 inputs the captured image data to learning model 2M and identifies the lesion detection result to be output. The image data input to learning model 2M may be image data obtained by performing various processes on the captured image data, or may be endoscopic image data. When a lesion detection result indicating the presence of a lesion is obtained and it is determined that a lesion has been detected (Step S14: YES), control unit 21 determines that the second start condition is satisfied, and proceeds with the process to start acquiring high-quality video data in Step S16. On the other hand, if a lesion detection result indicating the absence of a lesion is obtained and it is determined that no lesion is detected (step S14: NO), the control unit 21 determines that the second start condition is not satisfied, returns the processing to step S13, and waits until the start condition is satisfied.

[0061] The control unit 21 generates a sub-process and performs the process of step S15 in parallel with the process of step S14. The control unit 21 may synchronize the processes by performing inter-process communication between these two processes. The control unit 21 determines whether the movement amount of the endoscope 1 is equal to or less than a predetermined value as a third start condition (step S15). The control unit 21 executes a process of outputting information indicating the movement amount detection result by the movement amount detection unit 214 based on the captured image data. The control unit 21 determines whether the movement amount of the acquired movement amount detection result is larger or smaller than a preset threshold value. If it is determined that the movement amount is equal to or less than the predetermined value (step S15: YES), the control unit 21 determines that the third start condition is satisfied, and proceeds to the process of starting acquisition of high-quality video data in step S16. On the other hand, if it is determined that the movement amount is not equal to or less than the predetermined value (step S15: NO), the control unit 21 determines that the third start condition is not satisfied, and returns the process to step S13 to wait until the start condition is satisfied.

[0062] If it is determined that any one of the first to third start conditions is satisfied, the control unit 21 starts acquiring high-quality video data (step S16). The control unit 21 acquires captured image data for a predetermined time period set by, for example, accepting input from an operator from the timing of acquisition start. The acquisition time of the high-quality video data is acquired. The control unit 21 performs predetermined image processing on the acquired captured image data, generates high-quality video compressed by a predetermined video compression method, and acquires it as high-quality video data. The control unit 21 associates the high-quality video data with subject identification information and stores it in the image DB (step S17).

[0063] The control unit 21 judges whether or not to end the process (step S18). For example, when the operator instructs to end the endoscopic examination, or when the endoscope 1 is removed from the endoscope processor 2, the control unit 21 judges to end the process. When it is judged not to end the process (step S18: NO), the control unit 21 returns the process to step S12 and continues acquiring captured image data. In this case, the control unit 21 may acquire the next captured image data at the timing when acquisition of a high-quality moving image of a predetermined length of time is completed. Alternatively, the control unit 21 may acquire the next captured image data before acquisition of the high-quality moving image is completed, and proceed with the process of judging the start condition. When it is judged to end the process (step S18: YES), the control unit 21 terminates the series of processes.

[0064] The process sequences described in the above embodiments are not limited, and changes in the procedure may be allowed as long as they are not contrary to the nature of the sequences. For example, the execution order of each process step may be changed for the above process sequences, multiple process steps may be executed simultaneously, or the order of each process step may be changed each time a series of process sequences is executed. The control unit 21 may simultaneously execute the determination of the first to third start conditions described above. That is, the control unit 21 may execute the processes of steps S14 and S15 in parallel with the process of step S13.

[0065] In the above, the control unit 21 performs a judgment on all start conditions from the first start condition to the third start condition, but this embodiment is not limited to this. For example, the control unit 21 may receive a selection of which start condition from the first start condition to the third start condition to judge, and make a judgment only on the selected start condition. The control unit 21 obtains whether or not a judgment is required for each start condition by receiving an input operation of a keyboard, a touch panel, or the like via the input unit 23. The control unit 21 executes a judgment process on whether or not the start condition is satisfied only for the selected start condition, and obtains a high-quality video when the selected start condition is satisfied.

[0066] Furthermore, the control unit 21 may be configured to receive a selection of a threshold value for the start condition for each examination, and acquire a high-quality video according to the start condition including the selected threshold value. For example, in determining the third start condition in step S15, the control unit 21 determines whether the threshold value is greater than the threshold value for the amount of movement previously selected and set by the operator. Furthermore, the control unit 21 may also determine the second start condition in step S14, including whether the threshold value is greater than the threshold value for the accuracy of lesion detection. In step S14, the control unit 21 acquires the presence or absence of a lesion and the accuracy of detection as the lesion detection result. The accuracy of detection is determined, for example, based on the score of the output information of the learning model 2M. When the lesion detection result indicates the presence of a lesion and the acquired accuracy of detection is greater than the preset threshold value, the control unit 21 determines that the second start condition is satisfied, and acquires a high-quality video. The operator can select the amount of information to be acquired depending on the diagnosis to be made at the time of the examination or at a later date.

[0067] According to this embodiment, in addition to conventional still images and low-quality videos, high-quality videos are acquired in the endoscope processor 2. The high-quality videos are recorded for a limited time period with a resolution equal to or higher than that of the still images, so that it is possible to limit the amount of data stored for one examination. Even when making a diagnosis at a later date, a doctor can make a diagnosis based on the still images recorded in high quality and the high-quality videos limited to the required time period, and therefore an efficient and highly accurate diagnosis can be made.

[0068] The high-quality video is acquired simultaneously with still images in response to an operator's instruction, and is also acquired automatically based on a predetermined start condition. This prevents images from being forgotten or lesions from being overlooked, reducing the probability of misdiagnosis. Furthermore, since it is possible to collect limited high-quality video that records target areas with a high probability of lesions at high resolution, it is possible to efficiently collect information that can be used, for example, as training data for a machine learning model that outputs information about lesions.

[0069] Second embodiment In the second embodiment, a configuration for providing a single video by linking multiple high-quality videos will be described. Differences between the second embodiment and the first embodiment will be described below. Since the configuration other than the configuration described below is the same as that of the endoscope system of the second embodiment, the same reference numerals will be used for the common configuration and detailed description thereof will be omitted.

[0070] 7 is a functional block diagram of the endoscope processor 2 in the second embodiment. In the second embodiment, the control unit 21 of the endoscope processor 2 is configured to further function as a connection unit 220 by executing a program 2P stored in the storage unit 22.

[0071] In the second embodiment, the determination unit 215 instructs the image processing unit 211 to start outputting captured image data at the timing when it is determined to acquire each image data such as still image data, low-quality video data, and high-quality video data, and issues a timestamp indicating the time when the captured image data is output. The determination unit 215 outputs the issued timestamp to each of the still image acquisition unit 217, low-quality video acquisition unit 218, and high-quality video acquisition unit 219. Note that the determination unit 215 may output the timestamp to the image processing unit 211, and the image processing unit 211 may output the timestamp to each of the still image acquisition unit 217, low-quality video acquisition unit 218, and high-quality video acquisition unit 219. The still image acquisition unit 217, low-quality video acquisition unit 218, and high-quality video acquisition unit 219 each assign a timestamp to each image file by storing the acquired timestamp in the image DB 222 in association with each image file. Alternatively, the still image acquisition unit 217, the low image quality video acquisition unit 218, and the high image quality video acquisition unit 219 may be configured to embed a timestamp in each image file to be generated, thereby adding a timestamp to each image file. Note that the time information is not limited to that contained in each image file, and it is sufficient that the time information is associated with each image and stored in the image DB 222. For example, data on the time information may be associated with the image file and stored in the image DB 222.

[0072] Furthermore, the determination unit 215 outputs information indicating a start condition on which the determination is based, together with an instruction to start outputting the high-quality video data, to the high-quality video acquisition unit 219. The high-quality video acquisition unit 219 stores the acquired information indicating the start condition in the storage unit 22 in association with the high-quality video data.

[0073] The connection unit 220 connects a plurality of high-quality video data captured on the same subject to generate one connected video data. The connection unit 220 refers to the image DB 222 and acquires a plurality of high-quality video data associated with the same subject identification information. The connection unit 220 arranges the plurality of high-quality video data in chronological order based on time information associated with each acquired high-quality video data. Between high-quality video data adjacent in chronological order, the connection unit 220 connects the last frame of the high-quality video data at a previous time point with the first frame of the high-quality video data at a later time point to generate one connected video data.

[0074] In the case of connecting the preceding and following frames, the connecting unit 220 may perform the connection by interpolating between the preceding and following frames. For example, when a plurality of high-quality videos are acquired by capturing the same target location under the same starting conditions, it is preferable to perform frame interpolation of these plurality of videos to display them as smooth video data. The connecting unit 220 acquires high-quality video data associated with the same starting conditions from a plurality of high-quality video data acquired about the same subject. The connecting unit 220 performs frame interpolation between the preceding and following frames between the high-quality video data associated with the same starting conditions. In this case, the connecting unit 220 may perform frame interpolation when the acquisition time difference between the preceding and following frames is equal to or less than a predetermined value, such as 0.1 to 1 second. When the acquisition time interval between the preceding and following frames is greater than a predetermined value, it is highly likely that both frames do not include the same imaging target, so the connecting unit 220 may not interpolate frames.

[0075] The connection unit 220 performs, for example, an interpolation to generate an intermediate frame based on previous and next frames. Alternatively, the connection unit 220 may perform an interpolation to generate an intermediate frame based on still image data or low-quality video data. When still image data acquired at a time between previous and next frames is stored, the connection unit 220 may use the still image data as an intermediate frame. Alternatively, when a frame of low-quality video data acquired at a time between previous and next frames is stored, the connection unit 220 may use the frame of low-quality video data as an intermediate frame. The connection unit 220 may specify the still image data or low-quality video data to be used for the intermediate frame based on time information and subject identification information associated with the still image data or low-quality video data.

[0076] 8 is a flowchart showing an example of a connection process procedure executed by the endoscope processor 2. The control unit 21 of the endoscope processor 2 executes the following process, for example, based on the input contents from the input unit 23 connected to the endoscope processor 2. The control unit 21 may execute the following process when an examination is performed and new high-quality video data is stored.

[0077] The control unit 21 refers to the image DB 222 and acquires a plurality of high-quality video data associated with the same subject identification information (step S21). The control unit 21 arranges the plurality of high-quality video data in chronological order based on time information associated with each acquired high-quality video data.

[0078] The control unit 21 determines whether or not the high quality video data adjacent in time series were acquired based on the same start condition (step S22). If it is determined that the start conditions of the consecutive high quality video data do not match and are not the same start condition (step S22: NO), the control unit 21 skips the frame interpolation process of step S23.

[0079] When it is determined that the start conditions of the consecutive high quality video data are the same (step S22: YES), the control unit 21 interpolates frames (step S23). Specifically, the control unit 21 interpolates the last frame of the high quality video data at a previous time point and the first frame of the high quality video data at a later time point in the chronologically adjacent high quality video data. The control unit 21 executes the above process between multiple high quality video data that have the same start condition and are consecutive in chronological order.

[0080] The control unit 21 generates one piece of connected video data by connecting the last frame of the high-quality video data at a previous time point and the first frame of the high-quality video data at a later time point in each of the multiple high-quality video data arranged in chronological order associated with the same subject identification information (step S24). The control unit 21 displays the generated connected video data on the display device 3 via the output unit 24, and ends the series of processes. The control unit 21 may store the generated connected video data in the storage unit 22.

[0081] According to this embodiment, moving images acquired for the same subject are displayed as a series of linked moving images, which allows for smoother diagnosis. Furthermore, frame interpolation is performed for moving images that have the same starting conditions, allowing doctors to make diagnoses without noticing any missing frames.

[0082] In each of the above embodiments, the endoscope 1 is described as an example of a medical endoscope used on a human subject, but the endoscope 1 may also be an industrial endoscope used, for example, to inspect piping, etc.

[0083] In addition, the embodiments disclosed above are illustrative in all respects and should not be considered as limiting. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the scope of the claims and the scope equivalent to the claims. [Explanation of symbols]

[0084] 1 Endoscope 2 Endoscope processor 21 Control section 22 Memory section 2P Program 2A Recording Media 2M Learning Model 222 Image DB 211 Image processing section 212 Reception 213 Lesion Detection Unit 214 Movement amount detection unit 215 Judgment section 216 Identification information acquisition unit 217 Still image acquisition section 218 Low-quality video acquisition unit 219 High-quality video acquisition unit 220 Connection section

Claims

1. a determination unit that determines whether a start condition for starting acquisition of a moving image limited to a predetermined time length and having a resolution equal to or higher than that of a still image captured by an endoscope is satisfied; a moving image acquisition unit that starts acquiring the moving image when the determination unit determines that the start condition is satisfied; a reception unit that receives a start operation to instruct a start of acquisition of the moving image, The determination unit determines that the start condition is satisfied when a start operation is accepted by the acceptance unit, the start condition includes a start condition other than the start operation being accepted by the acceptance unit, The determination unit determines whether the other start condition is satisfied when the start operation is not accepted by the acceptance unit. Endoscope processor.

2. a lesion detection unit that detects a lesion in the still image, The determination unit determines that the start condition is satisfied when a lesion is detected by the lesion detection unit. The endoscope processor according to claim 1 .

3. a movement amount detection unit that detects a movement amount of the endoscope based on the still image, The determination unit determines that the start condition is satisfied when the movement amount detected by the movement amount detection unit is equal to or smaller than a predetermined value. The endoscope processor according to claim 1 or 2.

4. A determination unit that determines whether a start condition is satisfied to start acquisition of a moving image limited to a predetermined time length having a resolution equal to or higher than that of a still image captured by an endoscope; a moving image acquisition unit that starts acquiring the moving image when the determination unit determines that the start condition is satisfied; a movement amount detection unit that detects a movement amount of the endoscope based on the still image, The determination unit determines that the start condition is satisfied when the movement amount detected by the movement amount detection unit is equal to or smaller than a predetermined value. Endoscope processor.

5. A storage unit that stores the moving image and subject identification information of the subject photographed by the endoscope in association with each other. The endoscope processor according to any one of claims 1 to 4.

6. The storage unit stores the video in association with the start condition. The endoscope processor according to claim 5 .

7. a linking unit that links, in time series, a plurality of moving images associated with the same subject identification information stored in the storage unit; The endoscope processor according to claim 5 or 6.

8. When multiple moving images captured consecutively under the same starting condition are connected, the connection unit connects the multiple moving images by interpolating frames between the multiple moving images. The endoscope processor according to claim 7 .

9. a second moving image acquisition unit that acquires second moving images captured by the endoscope from the start to the end of an examination by the endoscope; The video has a higher resolution than the second video. The endoscope processor according to any one of claims 1 to 8.

10. The resolution of the video is high definition or higher. The endoscope processor according to any one of claims 1 to 9.

11. A processor for an endoscope, comprising: determining whether a start condition for starting acquisition of a moving image limited to a predetermined time length having a resolution equal to or higher than that of a still image captured by an endoscope is satisfied; When it is determined that the start condition is satisfied, the acquisition of the moving image is started; determining that the start condition is satisfied when a start operation for instructing the start of acquisition of the moving image is accepted; the start condition includes a start condition other than that the start operation has been accepted, If the start operation has not been accepted, it is determined whether the other start conditions are satisfied. An information processing method for performing a process.

12. A processor for an endoscope, comprising: determining whether a start condition for starting acquisition of a moving image limited to a predetermined time length having a resolution equal to or higher than that of a still image captured by an endoscope is satisfied; When it is determined that the start condition is satisfied, the acquisition of the moving image is started; The start condition is determined to be satisfied when the amount of movement of the endoscope detected based on the still image is equal to or smaller than a predetermined value. An information processing method for performing a process.

13. determining whether a start condition for starting acquisition of a moving image limited to a predetermined time length having a resolution equal to or higher than that of a still image captured by an endoscope is satisfied; When it is determined that the start condition is satisfied, the acquisition of the moving image is started; determining that the start condition is satisfied when a start operation for instructing the start of acquisition of the moving image is accepted; the start condition includes a start condition other than that the start operation has been accepted, If the start operation has not been accepted, it is determined whether the other start conditions are satisfied. A computer program that causes a computer to carry out processing.

14. A method for determining whether a start condition for starting acquisition of a moving image limited to a predetermined time length having a resolution equal to or higher than that of a still image captured by an endoscope is satisfied, When it is determined that the start condition is satisfied, the acquisition of the moving image is started; The start condition is determined to be satisfied when the amount of movement of the endoscope detected based on the still image is equal to or smaller than a predetermined value. A computer program that causes a computer to carry out processing.

Images