Image recording device, image recording method, and image recording program

Abstract

The image recording device comprises: an acquisition unit (51) for acquiring time series images of endoscopic examination; a lesion occurrence determination unit (52) for determining the occurrence of a lesion based on the acquired time series images; and a recording unit (53) for starting to record the time series images from the time point when the occurrence of the lesion is determined in the lesion occurrence determination unit (52).

Description

Technical Field

[0001] The present invention relates to an image recording device, an image recording method, and an image recording program. Background Art

[0002] In the medical field of diagnostic imaging, various systems have been developed to capture medical images of various anatomical structures associated with individual patients in order to classify and evaluate disease conditions. Known imaging systems include, for example, CT (computed tomography) systems, MRI (magnetic resonance imaging) systems, X-ray systems, ultrasound systems, and PET (positron emission tomography) systems.

[0003] Furthermore, Japanese Patent Application Publication No. 2007-528746 discloses a method for implementing a lesion detection function based on so-called computer-aided detection / diagnosis (CADe) by performing machine learning on annotation data provided by medical professionals such as doctors.

[0004] However, in an endoscopic examination, when lesion detection is performed using the aforementioned computer detection / diagnosis support (CADe), annotation data from medical professionals such as doctors do not necessarily include the time when a significant lesion first appeared.

[0005] Furthermore, in conventional methods, in order to reliably record the time when a lesion begins to appear in an endoscopic inspection image, all image data in the inspection must be recorded, which poses a problem of requiring a very large data recording device.

[0006] The present invention has been completed in view of the above situation, and its purpose is to provide an image recording device, an image recording method and an image recording program that can reliably and easily generate annotation data of the moment when lesions begin to appear in a time series inspection image when lesion detection is performed using computer detection / diagnosis support (CADe). Summary of the Invention

[0007] Means used to solve problems

[0008] An image recording device according to one embodiment of the present invention comprises: an acquisition unit for acquiring time-series images of an endoscopic examination; a buffer recording unit capable of temporarily recording all of the time-series images of the endoscopic examination; a lesion occurrence determination unit for acquiring the time-series images temporarily recorded in the buffer recording unit and determining the occurrence of a lesion associated with the acquired time-series images; and a recording unit for recording a specified image among the time-series images temporarily recorded in the buffer recording unit based on the occurrence of the lesion determined by the lesion occurrence determination unit.

[0009] An image recording method according to one embodiment of the present invention comprises: an acquisition step of acquiring time-series images of an endoscopic examination; a step of temporarily recording all of the time-series images of the endoscopic examination; a lesion occurrence determination step of acquiring the temporarily recorded time-series images and determining the occurrence of lesions associated with the acquired time-series images; and a recording step of recording specified images in the temporarily recorded time-series images based on the determined occurrence of the lesions.

[0010] An image recording program of one embodiment of the present invention causes a computer to execute the following steps: an acquisition step of acquiring time series images of an endoscopic examination; a step of temporarily recording all of the time series images of the endoscopic examination; a lesion occurrence determination step of acquiring the temporarily recorded time series images and determining the occurrence of lesions related to the acquired time series images; and a recording step of recording specified images in the temporarily recorded time series images based on the occurrence of the lesion determined in the lesion occurrence determination step. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a block diagram showing the configuration of an endoscope system including the image recording device according to the first embodiment of the present invention.

[0012] Figure 2 This is a block diagram illustrating the configuration of the image recording device and peripheral units according to the first embodiment.

[0013] Figure 3 This is a block diagram illustrating the configuration of a recording unit in the image recording apparatus according to the first embodiment.

[0014] Figure 4 This is a block diagram showing a more specific configuration of the image recording device according to the first embodiment.

[0015] Figure 5 This is a block diagram illustrating the configuration of an image recording device and peripheral components according to a second embodiment of the present invention.

[0016] Figure 6 This is a block diagram illustrating the configuration of an image recording device and peripheral units according to a third embodiment of the present invention.

[0017] Figure 7 This is a block diagram showing the configuration of an endoscope system including an image recording device according to a fourth embodiment of the present invention.

[0018] Figure 8 This is a block diagram illustrating the configuration of an image recording device and peripheral components according to a fourth embodiment. DETAILED DESCRIPTION

[0019] Hereinafter, embodiments of the present invention will be described using the drawings.

[0020] <First embodiment>

[0021] Figure 1 This is a block diagram showing the configuration of an endoscope system including the image recording device according to the first embodiment of the present invention.

[0022] like Figure 1 As shown, the endoscope system 1 of the present embodiment includes, for example, an endoscope 10 , a light source device 20 , a video processor 30 , an insertion shape detection device 40 , an image recording device 50 , a display device 60 , and an input device 70 .

[0023] The endoscope 10 includes an insertion portion 11 to be inserted into a subject, an operating portion 12 provided at the proximal end of the insertion portion 11, and a universal cord 13 extending from the operating portion 12. The endoscope 10 is detachably connected to the light source device 20 via a scope connector 13A provided at the end of the universal cord 13.

[0024] Furthermore, the endoscope 10 is configured to be detachably connected to the video processor 30 via an electrical connector 14A provided at the end of a cable 14 extending from a scope connector 13A. Furthermore, a light guide (not shown) for transmitting illumination light supplied from the light source device 20 is provided within the insertion portion 11, the operating portion 12, and the universal cord 13.

[0025] The insertion portion 11 is configured to have a flexible and elongated shape. Furthermore, the insertion portion 11 is configured to include, in order from the distal end side, a rigid distal end portion 11A, a bendable bending portion 11B, and a flexible elongated flexible tube portion 11C.

[0026] The distal end portion 11A is provided with an illumination window (not shown) for emitting illumination light, transmitted through a light guide provided within the insertion portion 11, toward a subject. Furthermore, the distal end portion 11A is provided with an imaging unit configured to operate in response to an imaging control signal supplied from the video processor 30, and to capture an image of the subject illuminated by the illumination light emitted through the illumination window and output an imaging signal. The imaging unit is configured to include an image sensor such as a CMOS image sensor or a CCD image sensor.

[0027] The operating portion 12 is configured to be operable by a user gripping it. Furthermore, the operating portion 12 includes an angle knob 121 configured to bend the bending portion 11B in four directions (up, down, left, and right) intersecting the longitudinal axis of the insertion portion 11: UDLR. Furthermore, the operating portion 12 includes one or more scope switches 122 configured to provide instructions corresponding to inputs from the operator (user), such as a release operation.

[0028] Here, the angle knob 121 and the scope switch 122 in the operation unit 12 function as an input device for external information.

[0029] The light source device 20 is configured to include, for example, one or more LEDs or one or more lamps as a light source. Furthermore, the light source device 20 is configured to emit illumination light for illuminating the interior of a subject inserted into the insertion portion 11 and to supply the illumination light to the endoscope 10. Furthermore, the light source device 20 is configured to vary the amount of illumination light in response to a system control signal supplied from the video processor 30.

[0030] The input device 70 is detachably connected to the video processor 30 via a cable 16 and includes one or more input interfaces such as a mouse, a keyboard, or a touch panel operated by an operator (user).

[0031] The input device 70 also functions as an external information input device for inputting a predetermined voice uttered by an operator. Furthermore, the input device 70 is configured to output instructions corresponding to an operation or voice input by an operator (user) to the video processor 30.

[0032] The insertion shape detection device 40 is detachably connected to the video processor 30 via the cable 15. Furthermore, in the present embodiment, the insertion shape detection device 40 is configured to detect a magnetic field generated by, for example, a source coil group provided in the insertion portion 11, and to obtain the position of each of the plurality of source coils included in the source coil group based on the intensity of the detected magnetic field.

[0033] Furthermore, the insertion shape detection device 40 is configured to calculate the insertion shape of the insertion portion 11 based on the positions of each of the plurality of source coils acquired as described above, and to generate insertion shape information indicating the calculated insertion shape and output it to the video processor 30 .

[0034] In this embodiment, each component of the insertion shape detection device 40 may be configured as an electronic circuit or as a circuit module within an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in this embodiment, for example, the insertion shape detection device 40 may be configured to include one or more processors (CPUs, etc.).

[0035] The display device (monitor) 60 is detachably connected to the video processor 30 via a cable 17 and includes, for example, a liquid crystal monitor. The display device 60 is configured to display an endoscopic image output from the video processor 30 on a screen.

[0036] The image recording device 50 is detachably connected to the video processor 30 via a cable 18. The image recording device 50 also includes an image acquisition unit that acquires a time series of endoscopic inspection images that have been appropriately processed by the video processor 30. The image recording device 50 also includes a recording unit that records the time series of endoscopic images, and a lesion presence determination unit that determines the presence of a lesion based on the acquired time series of endoscopic images. The image recording device 50 will be described in detail later.

[0037] The video processor 30 includes an image processing unit that receives the imaging signal output from the endoscope 10 and performs predetermined image processing to generate a time-series endoscopic image. Furthermore, the video processor 30 is configured to perform predetermined operations for displaying the generated endoscopic image on the display device 60. Furthermore, the video processor 30 is configured to generate and output various control signals for controlling the operation of the endoscope 10, the light source device 20, the image recording device 50, and the like.

[0038] Furthermore, as described above, the video processor 30 is configured to perform operations in accordance with instructions from the input device 70 and the scope switch 112 of the operation unit 12 , that is, operations based on external information.

[0039] For example, when a predetermined voice (such as lesion discovery) uttered by the operator is input to the input device 70 as described above, the video processor 30 transmits information corresponding to the operator's voice input as notification information regarding the operator's lesion discovery action to the lesion presence determination unit 52 of the image recording device 50. This will be described in detail later.

[0040] Furthermore, upon receiving, for example, release operation information or freeze operation information from the scope switch 112 of the operating unit 12, the video processor 30 transmits information as notification information of the operator's lesion discovery action to the lesion presence determination unit 52 of the image recording device 50 based on the operator's operation, as described above. This will be described in detail later.

[0041] Furthermore, the video processor 30 is configured to control the driving state of the insertion portion 11 based on the insertion shape information output from the insertion shape detection device 40 .

[0042] In addition, in this embodiment, the video processor 30 grasps the state of the insertion part 11 based on the insertion shape information output from the insertion shape detection device 40. For example, it can also detect the moving speed and moving direction of the insertion part 11 (for example, the withdrawal speed of the insertion part, etc.) to determine whether the insertion part 11 stops moving, changes in moving, or performs the treatment action of the endoscope.

[0043] Furthermore, the video processor 30 determines the movement of the above-mentioned endoscope insertion part based on the insertion shape information from the insertion shape detection device 40, and sends the determination result as movement stop information or movement change information of the endoscope insertion part, or movement information related to endoscopic treatment, to the lesion occurrence determination part (lesion discovery movement determination part 52A) in the image recording device 50.

[0044] In addition, in this embodiment, each component of the video processor 30 may be configured as an individual electronic circuit, or as a circuit module in an integrated circuit such as an FPGA (Field Programmable Gate Array). In addition, in this embodiment, for example, the video processor 30 may be configured to include one or more processors (CPU, etc.).

[0045] <Internal Structure of Image Recording Device 50 in First Embodiment>

[0046] Next, the configuration of the image recording device 50 according to this embodiment will be described in detail.

[0047] Figure 2 is a block diagram illustrating the configuration of the image recording device and peripheral components according to the first embodiment. Figure 3 1 is a block diagram illustrating the configuration of a recording unit in the image recording device according to the first embodiment. Figure 4 This is a block diagram showing a more specific configuration of the image recording device according to the first embodiment.

[0048] In this embodiment, each component of the image recording device 50 may be configured as an electronic circuit or as a circuit module within an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in this embodiment, for example, the image recording device 50 may be configured to include one or more processors (CPU, etc.).

[0049] like Figure 2 As shown, the image recording device 50 in the first embodiment includes: an image acquisition unit 51 connected to the video processor 30 and acquiring time series images of endoscopic examination; a lesion occurrence determination unit 52 that determines the occurrence of a lesion based on the time series images acquired in the image acquisition unit 51; and a recording unit 53 that starts recording the time series images from the time point when the occurrence of the lesion is determined in the lesion occurrence determination unit 52.

[0050] The image acquisition unit 51 acquires the imaging signal (image signal) output from the endoscope 10 after being subjected to predetermined image processing in the image processing unit of the video processor 30 , and sends the time-series image signal to the recording unit 53 and to the lesion occurrence identification unit 52 .

[0051] After receiving the time-series image signals, the lesion appearance identifying unit 52 identifies the appearance time of the lesion based on the time-series images and sends the information of the time to the recording unit 53 .

[0052] The recording unit 53 receives the time series images from the image acquisition unit 51, and on the other hand, receives the lesion occurrence time information from the lesion occurrence determination unit 52, and starts recording the time series images corresponding to the time information, for example, starts recording the lesion interval including the time immediately after the lesion occurs related to the time series images of the endoscopic examination.

[0053] Thus, in this embodiment, the recording unit 53 records time-series images corresponding to the time information from the lesion appearance determination unit 52. In other words, the image recording device of this embodiment can easily generate annotation data indicating the time when a lesion begins to appear in important images during lesion detection using computer detection / diagnosis support (CADe).

[0054] The recording unit 53 also includes a training data generating unit that generates training data for machine learning based on the annotation data corresponding to the time series images associated with the lesion identified as present by the lesion presence identifying unit 52. Furthermore, the recording unit 53 includes a learning model generating unit that generates a learning model for recognition based on the training data generated by the training data generating unit.

[0055] Here, the learning model functions as a recognition machine that recognizes the boundary lines of characteristic portions estimated to be lesions on the endoscopic image based on the generated teacher data.

[0056] The recording unit 53 also includes a recognition unit that performs a predetermined recognition process based on the learning model generated by the learning model generation unit. More specifically, the recording unit 53 generates a recognition algorithm in the recognition unit using a method such as so-called deep learning, and then appropriately performs the recognition process using this recognition algorithm.

[0057] In addition, in the first embodiment, the recording unit 53 is as follows Figure 3 The buffer recording unit 54 is shown. The recording unit 53 also appropriately records the time-series images from the image acquisition unit 51 in parallel with the recording of the time-series images corresponding to the timing information from the lesion appearance identification unit 52 described above.

[0058] The buffer recording unit 54 is capable of recording all of the time series images related to the endoscopic examination output from the image acquisition unit 51. Furthermore, in this embodiment, the time series images can be recorded starting from at least a predetermined time after the "time point at which the occurrence of the lesion is determined" in the lesion occurrence determination unit 52.

[0059] In addition, the lesion presence identification unit 52 can also acquire the time series images recorded in the buffer recording unit 54 and identify the “presence of a lesion” based on the acquired time series images.

[0060] At this time, the recording unit 53 can, for example, record a specified image in the time series image recorded by the buffer recording unit 54 based on the time of occurrence of the lesion determined by the lesion occurrence determination unit 52 after the end of the endoscopic examination, such as an interval (lesion interval) including the time immediately after the lesion appeared in the acquired image.

[0061] Next, refer to Figure 4 The specific configuration of the lesion occurrence identifying unit 52 in the image recording apparatus 50 according to the first embodiment will be described.

[0062] In addition, Figure 4 Although not particularly shown in the figure, the recording unit 53 includes the buffer recording unit 54 as described above. The recording unit 53 also includes the above-mentioned teacher data generating unit, learning model generating unit, and recognition unit.

[0063] Such as Figure 4 As shown, the lesion appearance specifying unit 52 in the first embodiment includes a lesion discovery operation determination unit 52A and an appearance time determination unit 52D that determines the time of appearance of a lesion.

[0064] The lesion discovery action determination unit 52A detects the lesion discovery action based on external information. Specifically, the lesion discovery action determination unit 52A functions as an operator notification information receiving unit. This unit receives notification information related to the operator's lesion discovery action output from the video processor 30 and detects the discovery of a lesion based on this notification information.

[0065] Specifically, the lesion discovery action determination unit 52A detects the discovery of a lesion based on the operator's voice information, predetermined release operation information related to the operator, or predetermined freeze operation information related to the operator transmitted from the video processor 30 .

[0066] As described above, the operator's voice information, release operation information, etc. are information sent to the lesion finding action determination unit 52A from the video processor 30 that receives external information from the input device 70 or the scope switch 112 of the operation unit 12, for example.

[0067] On the other hand, the lesion discovery operation determination unit 52A detects discovery of a lesion based on operation stop information or operation change information of the endoscope insertion portion sent from the video processor 30 or operation information related to endoscopic treatment.

[0068] As described above, the motion information related to the endoscope is, for example, motion information related to the insertion portion of the endoscope determined by the video processor 30 based on the insertion shape information from the insertion shape detection device 40 .

[0069] In this embodiment, when the lesion discovery action determination unit 52A detects an action of discovering a lesion, the appearance time determination unit 52D determines the time when the lesion appeared and sends time information based on the time to the recording unit 53 .

[0070] Specifically, when the occurrence time determination unit 52D detects the operator's action of discovering the lesion (the operator's voice, release operation, action of the endoscope insertion part, etc.), it sends the time information to the recording unit 53, so that the endoscopic image of the time series is recorded from the moment immediately after the detection, or from the moment traced back a specified time relative to the detection.

[0071] On the other hand, the recording unit 53 temporarily buffers and records the time-series images from the image acquisition unit 51 as described above, and also records the time-series images corresponding to the time information from the lesion appearance identification unit 52 .

[0072] <Function of the First Embodiment>

[0073] Next, the operation of the image recording device according to the first embodiment will be described.

[0074] First, the image acquisition unit 51 acquires time-series images related to endoscopy sent from the video processor 30 , and then sends the time-series image signals to the recording unit 53 and to the lesion presence identification unit 52 .

[0075] The recording unit 53 appropriately records the time-series images received from the image acquisition unit 51 in the buffer recording unit 54 , and waits for reception of time information from the lesion appearance identification unit 52 .

[0076] On the other hand, in the lesion occurrence determination unit 52, after receiving the time series image signal, the occurrence time of the lesion is determined based on the time series image through the above-mentioned lesion discovery action determination unit 52A and the occurrence time determination unit 52D, and the information of the time is sent to the recording unit 53.

[0077] The recording unit 53 receives the time series images from the image acquisition unit 51 and caches them in the buffer recording unit 54. On the other hand, when receiving the lesion occurrence time information from the lesion occurrence determination unit 52, it starts recording the time series images corresponding to the time information, for example, starts recording the lesion interval including the time immediately after the lesion occurrence related to the time series image of the endoscopic examination.

[0078] That is, the recording unit 53 records the time series images corresponding to the time information from the lesion occurrence determination unit 52, in other words, generates annotation data of the time when the lesion begins to appear in the important image in lesion detection using computer detection / diagnosis support (CADe).

[0079] Thereafter, the recording unit 53 generates teacher data for machine learning based on the generated annotation data, and further generates a learning model for recognition based on the teacher data.

[0080] Furthermore, the recording unit 53 generates an algorithm for recognition processing by a method such as so-called deep learning based on the generated learning model, and appropriately executes recognition processing.

[0081] <Effects of the First Embodiment>

[0082] In the image recording device of the first embodiment, during endoscopic examination, lesion detection results and lesion identification results are notified in real time by a computer, so even doctors with little experience can perform examinations at a level comparable to that of experienced doctors.

[0083] <Second embodiment>

[0084] Next, a second embodiment of the present invention will be described.

[0085] The image recording device of this second embodiment differs from that of the first embodiment in the structure of the lesion presence determination unit 52. In the image recording device of the first embodiment, the lesion presence determination unit 52 determines the time information recorded by the recording unit 53 based on the operator's lesion discovery action or the movement of the endoscope insertion unit. In other words, in the first embodiment, the time information is determined based on the operator's action.

[0086] In contrast, in the second embodiment, it is characterized in that the area of ​​interest of the lesion is automatically detected based on the image signal of the endoscopic examination image, the lesion is calculated based on the detection of the area of ​​interest, and specified time information corresponding to the calculated lesion information is determined, and further, based on the time information, a specified time series image is automatically recorded in the recording unit 53.

[0087] The other structures are the same as those of the first embodiment, and therefore, only the differences from the first embodiment will be described here, and the description of the common parts will be omitted.

[0088] <Internal Structure of Image Recording Device 50 in Second Embodiment>

[0089] Next, the configuration of the image recording device 50 according to the second embodiment will be described in detail.

[0090] Figure 5 This is a block diagram illustrating the configuration of an image recording device and peripheral components according to a second embodiment of the present invention.

[0091] In addition, Figure 5 Although not particularly shown in the figure, the recording unit 53 also includes the buffer recording unit 54 as described above. In addition, the recording unit 53 includes the above-mentioned teacher data generation unit, learning model generation unit, and recognition unit.

[0092] Furthermore, in this second embodiment, each component of the image recording device 50 may be configured as an electronic circuit or as a circuit module within an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in this embodiment, for example, the image recording device 50 may be configured to include one or more processors (CPUs, etc.).

[0093] like Figure 5As shown, the image recording device 50 of the second embodiment is the same as the first embodiment, and comprises: an image acquisition unit 51 connected to the video processor 30 and acquiring time series images of endoscopic examination; a lesion occurrence determination unit 52 that determines the occurrence of a lesion based on the time series images acquired in the image acquisition unit 51; and a recording unit 53 that starts recording the time series images from the time point when the occurrence of the lesion is determined in the lesion occurrence determination unit 52.

[0094] In the second embodiment, the image acquisition unit 51 also acquires the camera signal (image signal) output from the endoscope 10 after the prescribed image processing is performed in the image processing unit of the video processor 30, similar to the first embodiment, and sends the time series image signal to the recording unit 53 and to the lesion occurrence determination unit 52.

[0095] Furthermore, after receiving the time-series image signals, the lesion appearance determination unit 52 determines the time of lesion appearance based on the time-series images and transmits the time information to the recording unit 53. Furthermore, the recording unit 53 receives the time-series images from the image acquisition unit 51 and, on the other hand, receives the lesion appearance time information from the lesion appearance determination unit 52, and begins recording the time-series images corresponding to the time information. For example, it begins recording the lesion interval including the time immediately after the lesion appearance associated with the time-series images of the endoscopic examination.

[0096] As described above, in the second embodiment, annotation data of the time when a lesion starts to appear in an important image can also be easily generated in lesion detection using computer detection / diagnosis support (CADe).

[0097] Furthermore, in the second embodiment, the recording unit 53 also has the same teacher data generation unit, learning model generation unit, and recognition unit as the first embodiment, and uses a recognition algorithm generated by a method such as so-called deep learning to appropriately perform recognition processing.

[0098] In addition, in the second embodiment, the recording unit 53 also includes Figure 3 The buffer recording unit 54 shown appropriately records the time-series images from the image acquisition unit 51 in parallel with the recording of the time-series images corresponding to the timing information from the lesion appearance identification unit 52 .

[0099] like Figure 5 As shown, in the second embodiment, the lesion appearance specifying unit 52 includes a lesion detecting unit 52B, a lesion information calculating unit 52C, and an appearance time determining unit 52D.

[0100] The lesion detection unit 52B receives the endoscopic image signals, which are time-series images acquired by the image acquisition unit 51 , and performs known image processing on the image signals to detect a region of interest (ROI) as a “lesion”.

[0101] The lesion information calculation unit 52C calculates the region of interest (ROI) information related to the “lesion” based on the image data of the region of interest (ROI) detected by the lesion detection unit 52B.

[0102] Information related to the "lesion" includes, for example, time series information related to the region of interest (ROI) detected in the lesion detection unit 52B, position information related to the region of interest (ROI), distance information related to the region of interest (ROI), and size information related to the region of interest (ROI).

[0103] Specifically, the lesion information calculation unit 52C compares the time series changes, position, distance or size information of the region of interest (ROI) related to the "lesion" detected in the lesion detection unit 52B with the pre-calculated information, and judges it as a "lesion" under specified circumstances.

[0104] In this second embodiment, when a specified region of interest (ROI) is detected in the lesion detection unit 52B, the occurrence time determination unit 52D determines the time when the lesion appears based on the information related to the "lesion" calculated in the lesion information calculation unit 52C, and sends the time information based on the time to the recording unit 53.

[0105] <Function of the Second Embodiment>

[0106] Next, the operation of the image recording device according to the second embodiment will be described.

[0107] In the second embodiment, first, the image acquisition unit 51 acquires time-series images related to endoscopic examination sent from the video processor 30 , sends the time-series image signals to the recording unit 53 , and sends them to the lesion occurrence identification unit 52 .

[0108] The recording unit 53 appropriately records the time-series images received from the image acquisition unit 51 in the buffer recording unit 54 , and waits for reception of time information from the lesion appearance identification unit 52 .

[0109] On the other hand, the lesion detection unit 52B receives the time series images, ie, the endoscopic image signals acquired by the image acquisition unit 51, and performs known image processing on the image signals to detect a region of interest (ROI) as a "lesion".

[0110] Then, the lesion information calculation unit 52C calculates the “lesion” based on the image data of the region of interest (ROI) detected by the lesion detection unit 52B.

[0111] When a specified region of interest (ROI) is detected in the lesion detection unit 52B, the occurrence time determination unit 52D determines the time when the lesion appears based on the information related to the "lesion" calculated in the lesion information calculation unit 52C, and sends the time information based on the time to the recording unit 53.

[0112] The recording unit 53 receives the time series images from the image acquisition unit 51 and caches them in the buffer recording unit 54. On the other hand, when receiving the lesion occurrence time information from the lesion occurrence determination unit 52, it starts recording the time series images corresponding to the time information, for example, starts recording the lesion interval including the time immediately after the lesion occurrence related to the time series image of the endoscopic examination.

[0113] In this second embodiment, the recording unit 53 also records time series images corresponding to the time information from the lesion occurrence determination unit 52 (occurrence time determination unit 52D), in other words, generates annotation data of the time when the lesion begins to appear in the important image in lesion detection using computer detection / diagnosis support (CADe).

[0114] Then, the recording unit 53 generates teacher data for machine learning based on the generated annotation data, and further generates a learning model for recognition based on the teacher data.

[0115] Furthermore, the recording unit 53 generates an algorithm for recognition processing by a method such as so-called deep learning based on the generated learning model, and appropriately executes recognition processing.

[0116] <Effects of the Second Embodiment>

[0117] In the image recording device of the second embodiment, it is possible to automatically detect the area of ​​interest of the lesion based on the image signal of the endoscopic examination image, calculate the lesion based on the detection of the area of ​​interest, and determine the specified time information corresponding to the calculated lesion information. In addition, based on the time information, the specified time series images are automatically recorded in the recording unit, thereby enabling higher quality inspections.

[0118] <Third embodiment>

[0119] Next, a third embodiment of the present invention will be described.

[0120] The image recording device of this third embodiment is characterized in that when an area of ​​interest related to a lesion is detected in the lesion detection unit, a prescribed lesion matching process is performed on the image signal of the time series endoscopic inspection image cached in the buffer recording unit 54 of the recording unit 53, and prescribed time information is determined based on the result of the matching.

[0121] The other structures are the same as those of the first embodiment, and therefore, only the differences from the first embodiment will be described here, and the description of the common parts will be omitted.

[0122] <Internal Structure of Image Recording Device 50 in Third Embodiment>

[0123] Next, the configuration of the image recording device 50 according to the third embodiment will be described in detail.

[0124] Figure 6 This is a block diagram illustrating the configuration of an image recording device and peripheral units according to a third embodiment of the present invention.

[0125] Furthermore, in this third embodiment, each component of the image recording device 50 may be configured as an electronic circuit or as a circuit module within an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in this embodiment, for example, the image recording device 50 may be configured to include one or more processors (CPUs, etc.).

[0126] like Figure 6 As shown, the image recording device 50 in the third embodiment is the same as the first embodiment, and comprises: an image acquisition unit 51 connected to the video processor 30 and acquiring time series images of endoscopic examination; a lesion occurrence determination unit 52 for determining the occurrence of a lesion based on the time series images acquired in the image acquisition unit 51; and a recording unit 53, which starts recording the time series images from the time point when the occurrence of the lesion is determined in the lesion occurrence determination unit 52.

[0127] In addition, if Figure 6 As shown, in the third embodiment, the lesion appearance specifying unit 52 includes a lesion detecting unit 52B, a lesion matching unit 52E, and an appearance time determining unit 52D.

[0128] In the third embodiment, the image acquisition unit 51, similar to the first embodiment, acquires the video signal (image signal) output from the endoscope 10 after the prescribed image processing is performed in the image processing unit of the video processor 30, sends the time series image signal to the recording unit 53 (buffer recording unit 54), and sends it to the lesion detection unit 52B of the lesion occurrence determination unit 52.

[0129] Furthermore, the buffer recording unit 54 of the recording unit 53 stores the time-series endoscopic image signals received from the image acquisition unit 51 as a buffer.

[0130] As in the second embodiment, the lesion detection unit 52B receives the endoscopic image signals, which are time-series images acquired by the image acquisition unit 51, and performs known image processing on the image signals to detect a region of interest (ROI) as a "lesion".

[0131] When a region of interest (ROI) is detected in the lesion detection unit 52B, the lesion matching unit 52E obtains the endoscopic inspection image cached in the buffer recording unit 54 of the recording unit 53, and performs matching between the obtained image and the region of interest (ROI) detected in the lesion detection unit 52B.

[0132] In this third embodiment, when a specified region of interest (ROI) is detected in the lesion detection unit 52B, the occurrence time determination unit 52D determines the time when the lesion appears based on the matching result in the lesion matching unit 52E, and sends the time information based on the time to the recording unit 53.

[0133] <Function of the Third Embodiment>

[0134] Next, the operation of the image recording device according to the third embodiment will be described.

[0135] In the third embodiment, first, the image acquisition unit 51 acquires time-series images related to endoscopic examination sent from the video processor 30, sends the time-series image signals to the buffer recording unit 54 of the recording unit 53, and sends them to the lesion occurrence determination unit 52 (lesion detection unit 52B).

[0136] The recording unit 53 appropriately records the time-series images received from the image acquisition unit 51 in the buffer recording unit 54 , and waits for reception of time information from the lesion appearance identification unit 52 .

[0137] On the other hand, the lesion detection unit 52B receives the endoscopic image signals, which are time-series images acquired by the image acquisition unit 51, and performs known image processing on the image signals to detect a region of interest (ROI) as a "lesion".

[0138] In addition, when a region of interest (ROI) is detected in the lesion detection unit 52B, the lesion matching unit 52E obtains the endoscopic inspection image cached in the buffer recording unit 54 and performs matching between the obtained image and the region of interest (ROI) detected in the lesion detection unit 52B.

[0139] Then, the appearance time determination unit 52D determines the time when the lesion appeared based on the matching result of the lesion matching unit 52E, and sends time information based on the time to the recording unit 53 .

[0140] The recording unit 53 receives the time series images from the image acquisition unit 51 and caches them in the buffer recording unit 54. On the other hand, when receiving the lesion occurrence time information from the lesion occurrence determination unit 52, it starts recording the time series images corresponding to the time information, for example, starts recording the lesion interval including the time immediately after the lesion occurrence related to the time series image of the endoscopic examination.

[0141] In this third embodiment, the recording unit 53 records a time series image corresponding to the moment information from the lesion occurrence determination unit 52 (occurrence time determination unit 52D), in other words, generates annotation data of the moment when the lesion begins to appear in an important image in lesion detection using computer detection / diagnosis support (CADe).

[0142] Then, the recording unit 53 generates teacher data for machine learning based on the generated annotation data, and further generates a learning model for recognition based on the teacher data.

[0143] Furthermore, the recording unit 53 generates an algorithm for recognition processing by a method such as so-called deep learning based on the generated learning model, and appropriately executes recognition processing.

[0144] <Effects of the Third Embodiment>

[0145] In the image recording device of the third embodiment, when a region of interest related to a lesion is detected in the lesion detection unit, a prescribed lesion matching process is performed on the image signal of the time series endoscopic inspection image cached in the buffer recording unit 54 of the recording unit 53, and prescribed time information is determined based on the result of the matching, thereby enabling a higher quality inspection.

[0146] <Fourth embodiment>

[0147] Next, a fourth embodiment of the present invention will be described.

[0148] In the first to third embodiments, the image recording device 50 is provided separately from the video processor 30. However, the fourth embodiment is characterized in that the function of the image recording device 50 is provided within the video processor.

[0149] The other structures are the same as those of the first embodiment, and therefore only the differences from the first embodiment will be described here, and description of the common parts will be omitted.

[0150] Figure 7is a block diagram showing the configuration of an endoscope system including an image recording device according to a fourth embodiment of the present invention. Figure 8 This is a block diagram illustrating the configuration of an image recording device and peripheral components according to a fourth embodiment.

[0151] like Figure 7 As shown, in the fourth embodiment, it is characterized in that an image recording device 50 having the same structure as the image recording device 50 of the first embodiment is provided in the video processor 130, and the video processor 130 has an image processing unit 31 that performs the same image processing as the video processor 30 of the first embodiment.

[0152] The image recording device 50 of the fourth embodiment is the same as that of the first embodiment except that it is integrated with the video processor, and its operation and effects are also the same as those of the first embodiment.

[0153] In the fourth embodiment, each component of the image recording device 50 may be configured as an electronic circuit or as a circuit module within an integrated circuit such as an FPGA (Field Programmable Gate Array). Furthermore, in the present embodiment, for example, the image recording device 50 may be configured to include one or more processors (CPUs, etc.).

[0154] The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit of the present invention.

Claims

1. An image recording device, It is characterized in that have: an acquisition unit that acquires time-series images of endoscopy; a buffer recording unit capable of temporarily recording the time series images of the endoscope examination; a lesion appearance determination unit that acquires the time series images temporarily recorded in the buffer recording unit and determines an appearance time point of a lesion associated with the acquired time series images; as well as a recording unit that records, as non-temporary records, the time series images temporarily recorded in the buffer recording unit, the time after a predetermined time after the occurrence of the lesion. The lesion occurrence determination unit includes a lesion discovery action determination unit and an occurrence time determination unit. The lesion discovery action determination unit detects the operator's lesion discovery action based on information from the outside. When the lesion discovery action determination unit detects the lesion discovery action, the occurrence time determination unit determines the occurrence time point of the lesion. The lesion discovery action determination unit has an operator notification information receiving unit, which detects the discovery of lesions based on notification information related to the operator's lesion discovery action, or the lesion discovery action determination unit has an endoscope action information receiving unit, which detects the discovery of lesions based on information related to the action of the endoscope.

2. The image recording device according to claim 1, It is characterized in that The recording unit records a section including a time immediately after the lesion appears in the time series images temporarily recorded in the buffer recording unit.

3. The image recording device according to claim 1, It is characterized in that The lesion occurrence determination unit sends time information to the recording unit so that the time series of endoscopic images is recorded from a time after a predetermined time relative to when the lesion discovery action determination unit detects an action of discovering the lesion, The recording unit records time-series images corresponding to the time information.

4. The image recording device according to claim 1, It is characterized in that The operator notification information receiving unit detects the discovery of a lesion based on the operator's utterance information, predetermined release operation information related to the operator, or predetermined freeze operation information related to the operator.

5. The image recording device according to claim 1, It is characterized in that The endoscope operation information receiving unit detects the discovery of a lesion based on operation stop information or operation change information of an endoscope insertion unit or operation information related to an endoscopic treatment.

6. The image recording device according to claim 1, It is characterized in that The image recording device comprises: a teacher data generating unit that generates teacher data for machine learning based on annotation data corresponding to the time series images related to the lesion determined to have appeared by the lesion appearance determining unit; as well as A learning model generating unit generates a learning model for identification based on the teacher data generated by the teacher data generating unit.

7. The image recording device according to claim 6, It is characterized in that The image recording device further includes a recognition unit that performs a predetermined recognition process based on the learning model generated by the learning model generation unit.

8. An image recording method, It is characterized in that have: An acquisition step of acquiring time series images of endoscopy; a step of temporarily recording said time series images of said endoscopic examination; a lesion occurrence determination step of acquiring the temporarily recorded time series images and determining the occurrence time point of the lesion associated with the acquired time series images; as well as a recording step of recording, as a non-temporary record, a record of a time after a predetermined time from a time point of occurrence of the lesion in the temporarily recorded time series images; In the lesion occurrence determination step, the operator's action of discovering the lesion is detected based on information from the outside, and when the action of discovering the lesion is detected, the occurrence time point of the lesion is determined. Among them, the discovery of the lesion is detected based on the notification information related to the lesion discovery action of the operator, or the discovery of the lesion is detected based on the information related to the action of the endoscope.

Images