Medical support systems and medical support methods

Abstract

A first mark-imparting unit (102) acquires presence information showing the presence of a site of biopsy in an endoscopic image and site information showing a site included in the endoscopic image, and imparts a first mark showing that the biopsy has been performed or the biopsy should be performed on the basis of the presence information and the site information to a position corresponding to the site of biopsy in an organ model image. A display image production unit (100) displays the organ model image in which the first mark has been imparted on a display device (12b).

Description

Technical Field

[0001] The present disclosure relates to a medical support system and a medical support method for marking organ model images such as schema diagrams.

Background Art

[0002] In endoscopy, a doctor observes an endoscopic image displayed on a display device. When a lesion is found, the doctor operates a release switch of the endoscope to capture (save) the endoscopic image of the lesion. At this time, the doctor may use a treatment tool called forceps to collect a part of the lesion tissue. Collecting a part of the lesion tissue in endoscopy is called "biopsy", and the lesion tissue collected by biopsy is sent for a pathological examination and used as material for a definitive diagnosis.

[0003] Patent Document 1 discloses a reporting system that allows a doctor to mark the sampling position of a specimen on an endoscopic image taken during an upper endoscopy and mark the sampling position of the specimen on a schema diagram that schematically represents the shape of the observed organ during a lower endoscopy.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In preparing reports for biopsy examinations, physicians mark the endoscopic images of the biopsy site with a biopsy mark and attach these images to the report. At the same time, physicians also mark the corresponding biopsy site on the schematic diagram included in the report. By marking the biopsy site on the schematic diagram, other physicians can quickly identify the location of the biopsy site in that examination when viewing the diagram.

[0006] In current report creation processes, physicians manually add biopsy marks to both endoscopic images and schematic diagrams, which is time-consuming. Therefore, there is a need for the development of technology that can efficiently support report creation.

[0007] This disclosure is made in light of these circumstances, and its purpose is to provide a technology for automatically marking organ model images such as schematic diagrams. [Means for solving the problem]

[0008] To solve the above problems, a medical support system according to one aspect of the present invention comprises one or more processors having hardware, the one or more processors acquire presence information indicating the presence of a biopsy site in an endoscopic image and location information indicating a part included in the endoscopic image, and based on the presence information and location information, assigns a first mark indicating that a biopsy has been performed or should be performed to a position in an organ model image corresponding to the biopsy site, and displays the organ model image to which the first mark has been assigned.

[0009] Another aspect of the present invention is a medical support method comprising: acquiring presence information indicating the presence of a biopsy site in an endoscopic image; acquiring location information indicating a part included in the endoscopic image; applying a first mark indicating that a biopsy has been performed or should be performed, at a position corresponding to the biopsy site in an organ model image, based on the presence information and location information; and displaying the organ model image to which the first mark has been applied.

[0010] Furthermore, any combination of the above components, as well as conversions of the expressions of this disclosure between methods, apparatus, systems, recording media, computer programs, etc., are also valid forms of this disclosure. [Brief explanation of the drawing]

[0011] [Figure 1] This is a diagram showing the configuration of a medical support system according to an embodiment. [Figure 2] This diagram shows the functional blocks of a server device. [Figure 3] This diagram shows the functional blocks of an information processing device. [Figure 4] This figure shows an example of an organ model image. [Figure 5] This figure shows multiple subregions that make up an organ model image. [Figure 6] This figure shows an example of a report creation screen for entering test results. [Figure 7] This figure shows an example of a selection screen for selecting endoscopic images. [Figure 8] This figure shows an example of an endoscopic image editing screen. [Figure 9] This figure shows the endoscopic image with a second biopsy mark placed on it. [Figure 10] This figure shows an example of the schema diagram editing screen. [Figure 11] This figure shows an example of the editing screen for another endoscopic image. [Figure 12] This figure shows an example of the editing screen for another endoscopic image. [Figure 13] This figure shows an example of the schema diagram editing screen. [Figure 14] This diagram shows the estimated direction of movement of the endoscope. [Figure 15] This figure shows an example of an editing screen for endoscopic images, including organ model images. [Modes for carrying out the invention]

[0012] FIG. 1 shows the configuration of the medical support system 1 according to the embodiment. The medical support system 1 is provided in a medical facility such as a hospital that performs endoscopic examinations. In the medical support system 1, the server device 2, the image analysis device 3, the image storage device 8, the endoscope system 9, and the terminal device 10b are communicably connected via a network 4 such as a LAN (Local Area Network). The endoscope system 9 is provided in an examination room and includes an endoscope observation device 5 and a terminal device 10a. In the medical support system 1, the server device 2, the image analysis device 3, and the image storage device 8 may be provided outside the medical facility, for example, as a cloud server.

[0013] The endoscope observation device 5 is connected to an endoscope 7 that is inserted into a patient's digestive tract. The endoscope 7 has a light guide for transmitting illumination light supplied from the endoscope observation device 5 to illuminate the inside of the digestive tract. At the tip, there are provided an illumination window for emitting the illumination light transmitted by the light guide to the biological tissue, and an imaging unit that captures the biological tissue at a predetermined cycle and outputs an imaging signal to the endoscope observation device 5. The imaging unit includes a solid-state imaging device (for example, a CCD image sensor or a CMOS image sensor) that converts incident light into an electrical signal.

[0014] The endoscope observation device 5 performs image processing on the imaging signal photoelectrically converted by the solid-state imaging device of the endoscope 7 to generate an endoscope image, and displays it on the display device 6 in real time. The endoscope observation device 5 may be provided with a function of performing special image processing for the purpose of highlighting display, etc., in addition to normal image processing such as A / D conversion and noise removal. The endoscope observation device 5 generates an endoscope image at a predetermined cycle (for example, 1 / 60 seconds). The endoscope observation device 5 may be constituted by one or more processors having dedicated hardware, or may be constituted by one or more processors having general-purpose hardware. The endoscope 7 of the embodiment is a flexible endoscope and has a forceps channel for inserting an endoscope treatment tool. A doctor can insert a biopsy forceps into the forceps channel and operate the inserted biopsy forceps to perform a biopsy during an endoscopic examination and collect a part of the lesion tissue.

[0015] The doctor observes the endoscopic image displayed on the display device 6 in accordance with the inspection procedure. While moving the endoscope 7, the doctor observes the endoscopic image. When the biological tissue to be captured appears on the display device 6, the doctor operates the release switch of the endoscope 7. The endoscopic observation device 5 captures (saves) the endoscopic image at the timing when the release switch is operated, and transmits the captured endoscopic image to the image storage device 8 together with information (image ID) for identifying the endoscopic image. The endoscopic observation device 5 may assign an image ID including a serial number to the endoscopic image in the order of capture. Note that the endoscopic observation device 5 may transmit a plurality of captured endoscopic images to the image storage device 8 collectively after the inspection is completed. The image storage device 8 records the endoscopic images transmitted from the endoscopic observation device 5 in association with an inspection ID for identifying the endoscopic examination.

[0016] In the embodiment, "imaging" means the operation in which the solid-state imaging device of the endoscope 7 converts incident light into an electrical signal, and "capture" means the operation of saving (recording) the endoscopic image generated by the endoscopic observation device 5. Note that "imaging" may include the operation from the converted electrical signal until the endoscopic observation device 5 generates an endoscopic image.

[0017] The terminal device 10a includes an information processing device 11a and a display device 12a, and is provided in the examination room. The terminal device 10a may be used for a doctor, a nurse, etc. to confirm information regarding the biological tissue being imaged in real time during an endoscopic examination.

[0018] The terminal device 10b includes an information processing device 11b and a display device 12b, and is provided in a room other than the examination room. The terminal device 10b is used when a doctor creates a report of an endoscopic examination. In a medical facility, the terminal devices 10a and 10b may be configured by one or more processors having general-purpose hardware.

[0019] In the medical support system 1 of this embodiment, the endoscope observation device 5 displays the endoscopic image on the display device 6 in real time and supplies the endoscopic image, along with metadata of the image, to the image analysis device 3 in real time. Here, the metadata includes at least the frame number of the image and the time of acquisition, and the frame number may be information indicating which frame it is since the endoscope 7 started taking images.

[0020] Image analysis device 3 is a computer that analyzes endoscopic images, detects lesions contained in the endoscopic images, and performs a qualitative diagnosis of the detected lesions. Image analysis device 3 is a CAD (computer-aided diagnosis) system with AI (artificial intelligence) diagnostic capabilities. This may be the case. The image analysis device 3 may be composed of one or more processors having dedicated hardware, but it may also be composed of one or more processors having general-purpose hardware.

[0021] Image analysis device 3 utilizes a pre-trained model generated by machine learning using training endoscopic images, information indicating organs and regions included in the endoscopic images, and information regarding lesion areas included in the endoscopic images as training data. Endoscopic image annotation is performed by an annotator with specialized knowledge, such as a physician, and machine learning may use deep learning methods such as CNN, RNN, and LSTM. When this pre-trained model receives an endoscopic image as input, it outputs information indicating the captured organ, information indicating the captured region, and information regarding the captured lesion (lesion information). The lesion information output by image analysis device 3 includes at least lesion presence information indicating whether or not a lesion is present in the endoscopic image. If a lesion is present, the lesion information may include information indicating the size of the lesion, information indicating the location of the lesion's contour, information indicating the shape of the lesion, information indicating the depth of the lesion's invasion, and a qualitative diagnosis result of the lesion. The qualitative diagnosis result of the lesion includes the type of lesion. During endoscopic examination, image analysis device 3 receives endoscopic images in real time from the endoscopic observation device 5 and outputs information indicating the organ, information indicating the region, and lesion information for each endoscopic image. Hereafter, the information indicating organs, information indicating locations, and information regarding lesions, which are output for each endoscopic image, will be collectively referred to as "image analysis information."

[0022] When a user operates the release switch (capture operation), the endoscope observation device 5 provides the image analysis device 3 with information indicating that a capture operation has been performed (capture operation information), along with the frame number, acquisition time, and image ID of the captured endoscopic image. Upon receiving the capture operation information, the image analysis device 3 provides the server device 2 with the examination ID, image ID, frame number, acquisition time information, and image analysis information for the provided frame number. Here, the image ID, frame number, acquisition time information, and image analysis information constitute "additional information" that expresses the characteristics and properties of the endoscopic image. Upon receiving the capture operation information, the image analysis device 3 transmits the additional information along with the examination ID to the server device 2, and the server device 2 records the additional information linked to the examination ID.

[0023] When the user finishes the endoscopic examination, they operate the examination completion button on the endoscope observation device 5. The operation information of the examination completion button is supplied to the server device 2 and the image analysis device 3, and the server device 2 and the image analysis device 3 recognize the completion of the endoscopic examination.

[0024] Figure 2 shows the functional blocks of the server device 2. The server device 2 comprises a communication unit 20, a processing unit 30, and a storage device 60. The communication unit 20 transmits and receives information such as data and instructions to and from the image analysis device 3, the endoscope observation device 5, the image storage device 8, terminal devices 10a and 10b via the network 4. The processing unit 30 has an order information acquisition unit 40 and an additional information acquisition unit 42. The storage device 60 has an order information storage unit 62 and an additional information storage unit 64.

[0025] Server device 2 includes a computer, and the various functions shown in Figure 2 are realized by the computer executing a program. The computer includes, as hardware, memory for loading the program, one or more processors for executing the loaded program, auxiliary storage devices, and other LSIs. The processor is composed of multiple electronic circuits, including semiconductor integrated circuits and LSIs, and these multiple electronic circuits may be mounted on one chip or on multiple chips. The functional blocks shown in Figure 2 are realized through the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof.

[0026] The order information acquisition unit 40 acquires order information for endoscopic examinations from the hospital information system. For example, before the start of the day's examination operations at a medical facility, the order information acquisition unit 40 acquires the order information for the day from the hospital information system and stores it in the order information storage unit 62. Before the start of the examination, the endoscopy observation device 5 or the information processing device 11a may read the order information for the examination to be performed from the order information storage unit 62 and display it on the display device.

[0027] The additional information acquisition unit 42 acquires the examination ID and additional information of the endoscopic image from the image analysis device 3, and stores the additional information in the additional information storage unit 64, linked to the examination ID. The additional information of the endoscopic image includes the image ID, frame number, shooting time information, and image analysis information.

[0028] Figure 3 shows the functional blocks of the information processing device 11b. The information processing device 11b has functions to support the creation of reports for biopsy examinations and includes a communication unit 76, an input unit 78, a processing unit 80, and a storage device 120. The communication unit 76 transmits and receives information such as data and instructions to and from the server device 2, image analysis device 3, endoscope observation device 5, image storage device 8, and terminal device 10a via the network 4. The processing unit 80 includes an operation reception unit 82, an acquisition unit 84, a display screen generation unit 100, a first mark assignment unit 102, a second mark assignment unit 104, and a registration processing unit 106. The acquisition unit 84 includes an image acquisition unit 86 and an additional information acquisition unit 88. The storage device 120 includes an image storage unit 122, an additional information storage unit 124, and a schema image storage unit 126.

[0029] Figure 4 shows an example of an organ model image stored in the schematic image storage unit 126. The schematic image storage unit 126 stores schematic diagrams that schematically represent the shape of the observed organ as organ model images. Schematic diagrams are attached to the examination report to show the location of the biopsy site within the organ, and Figure 4 shows a schematic diagram of the stomach. The schematic image storage unit 126 may also store schematic diagrams of other organs observed in upper endoscopy, such as the esophagus and duodenum, and schematic diagrams of the large intestine in lower endoscopy.

[0030] Figure 5 shows multiple subregions that make up an organ model image. In this embodiment, the schematic diagram of the stomach is divided into multiple subregions. As described above, the trained model in the image analysis device 3 is machine-learned to output information indicating the captured area (the area included in the endoscopic image) when an endoscopic image is input, but it is preferable that the organ model image is divided into multiple subregions corresponding to the multiple areas output by the image analysis device 3. In other words, in this embodiment, one organ is divided into multiple areas, the schematic diagram of the organ is divided into multiple subregions, and one area of ​​the organ is associated with one subregion in the schematic diagram. In this embodiment, the organ "stomach" is divided into multiple areas: "fundus", "upper body looking up at lesser curvature", "upper body looking up at posterior wall", "mid body looking down at lesser curvature", "mid body looking up", "lower body looking down", "lower body looking up", "gastric angle looking down", "gastric angle looking up", "antrum", and "pyloric antrum", and the subregions shown in Figure 5 are set according to this division.

[0031] The information processing device 11b includes a computer, and the various functions shown in Figure 3 are realized by the computer executing a program. The computer includes, as hardware, a memory for loading the program, one or more processors for executing the loaded program, auxiliary storage devices, and other LSIs. The processor is composed of multiple electronic circuits, including semiconductor integrated circuits and LSIs, and these multiple electronic circuits may be mounted on one chip or on multiple chips. The functional blocks shown in Figure 3 are realized through the cooperation of hardware and software, and therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof.

[0032] After the endoscopic examination is completed, the user, who is a physician, enters their user ID and password into the information processing device 11b to log in. Once the user logs in, an application for creating the examination report is launched, and a list of completed examinations is displayed on the display device 12b. This list of completed examinations displays examination information such as patient name, patient ID, examination date and time, and examination items, and the user operates the input unit 78, such as a mouse or keyboard, to select the examination for which the report will be created. When the operation reception unit 82 receives the examination selection operation, the image acquisition unit 86 acquires multiple endoscopic images associated with the examination ID of the examination selected by the user from the image storage device 8 and stores them in the image storage unit 122, and the additional information acquisition unit 88 acquires additional information associated with the examination ID of the examination selected by the user from the server device 2 and stores it in the additional information storage unit 124. The display screen generation unit 100 generates the report creation screen and displays it on the display device 12b.

[0033] Figure 6 shows an example of a report creation screen for inputting test results. The report creation screen is displayed on the display device 12b with the report tab 54b selected. The upper part of the screen displays the patient's name, patient ID, date of birth, test items, test date, and information of the performing physician. This information is included in the test order information and may be obtained from the server device 2. In this embodiment, it is assumed that a biopsy has been performed for the test for which the report is to be created. The user operates the input unit 78 to select both the "Pathology Order Available" and "Schema Attached" checkboxes, and a check mark is displayed on both checkboxes to indicate that they have been selected.

[0034] The report creation screen consists of two areas: the left area is an attached image display area 56 for displaying attached endoscopic images, and the right area is an input area 58 for the user to input examination results. The input area 58 is provided for inputting diagnostic information for the observation range in upper endoscopy: "esophagus," "stomach," and "duodenum." The input area 58 may have a format that displays multiple options for examination results and allows the user to input the diagnostic information by selecting checkboxes, or it may have a free format for free text input.

[0035] The attached image display area 56 is an area for displaying endoscopic images to be attached to the report side by side. The user selects the endoscopic images to be attached to the report from the endoscopic image selection screen. The user selects the recorded image tab 54a to display a list of endoscopic images captured during the examination on the display device 12b.

[0036] Figure 7 shows an example of a selection screen for selecting endoscopic images to attach to a report. The endoscopic image selection screen is displayed on the display device 12b with the recorded image tab 54a selected. The display screen generation unit 100 generates a selection screen in which multiple endoscopic images captured by the user are arranged in the order in which they were taken, and displays it in the list display area 50. In the list display area 50, the endoscopic images may be displayed as reduced thumbnail images. The display screen generation unit 100 may refer to additional information for each endoscopic image and display the image ID and the part name indicating the part included in each endoscopic image together with the endoscopic image.

[0037] The endoscopic images displayed in the list display area 50 are provided with checkboxes. When a user moves the mouse to place the mouse pointer over a checkbox and left-clicks, the operation reception unit 82 receives this as an operation to select the endoscopic image as an attached image for the report, and the endoscopic image is selected as an attached image for the report. The endoscopic images selected as attached images for the report are displayed side by side in the attached image display area 56 (see Figure 6) when the report creation screen is displayed.

[0038] In the list display area 50, when a user places the mouse pointer over an endoscope image and performs a predetermined operation, the operation reception unit 82 receives this as an operation to enlarge the endoscope image, and the display screen generation unit 100 generates a display screen including the editably enlarged endoscope image and displays it on the display device 12b. The user operation to instruct the enlargement display may be a double-click operation on the endoscope image. In this case, the user double-clicks the endoscope image of image ID 3, which is a photograph of the biopsy site.

[0039] Figure 8 shows an example of an endoscopic image editing screen. When a user double-clicks the endoscopic image with image ID 3 in the list display area 50, the display screen generation unit 100 generates a display screen including an editable, enlarged endoscopic image 148a and displays it on the display device 12b. By enlarging the endoscopic image, the user can more easily confirm the biopsy site 150a included in the endoscopic image 148a. The biopsy site 150a may be lesion tissue photographed before the biopsy, or lesion tissue photographed after the biopsy. For example, if the physician captures the endoscopic image immediately before performing the biopsy, the biopsy site 150a is the location of the lesion tissue immediately before it is collected. If the physician captures the endoscopic image immediately after performing the biopsy, the biopsy site 150a is the location of the lesion tissue that has already been collected. The biopsy site 150a may also be a location that the physician has determined should be biopsied.

[0040] When the user presses the biopsy designation button 152, the second marking unit 104 becomes capable of adding a second biopsy mark to the enlarged endoscopic image 148a in response to the user's operation, indicating that a biopsy has been performed or should be performed. In the case of mouse operation, when the user places the mouse pointer near the biopsy site 150a included in the endoscopic image 148a and performs a predetermined operation, the operation reception unit 82 receives this as an operation to place the second biopsy mark at the mouse pointer's position. Therefore, the second biopsy mark indicates the location of the biopsy or the location that should be biopsied in the endoscopic image 148a.

[0041] Figure 9 shows the state in which the second biopsy mark 170a is placed on the endoscopic image 148a. When the operation reception unit 82 receives a user operation to place the second biopsy mark 170a on the endoscopic image 148a, the second mark placement unit 104 places the second biopsy mark 170a at the position specified by the user operation.

[0042] In the example shown in Figure 9, the second biopsy mark 170a "<1" is placed near the biopsy site 150a. The numbers included in the second biopsy marks indicate the order in which the user marked the biopsy sites; therefore, the first second biopsy mark placed will be "<1", the second second biopsy mark placed will be "<2", and so on. The second mark placement unit 104 counts the number of second biopsy marks placed and sets the second biopsy marks to be sequential numbers.

[0043] The information processing device 11b of this embodiment has a function that, when a second biopsy mark 170a is added to the endoscopic image 148a, automatically adds a first biopsy mark corresponding to the second biopsy mark 170a on the schematic diagram. The first biopsy mark corresponding to the second biopsy mark 170a is a biopsy mark that indicates that a biopsy has been performed or should be performed. Therefore, the first biopsy mark indicates the location where a biopsy was performed or should be performed on the schematic diagram. When a user adds a second biopsy mark 170a to the endoscopic image 148a, the information processing device 11b automatically adds a first biopsy mark on the schematic diagram, thereby supporting the user's efficient report creation work.

[0044] Specifically, when the second marking unit 104 applies a second biopsy mark 170a to the endoscopic image 148a, the first marking unit 102 acquires existence information indicating the presence of a biopsy site in the endoscopic image 148a, and site information indicating the part included in the endoscopic image 148a. The existence information may include information indicating the presence of a site to be biopsied in the endoscopic image 148a, or information indicating the presence of a site that has already been biopsied in the endoscopic image 148a. The existence information may also include information indicating that a second biopsy mark 170a has been applied to the endoscopic image 148a.

[0045] The first marking unit 102 acquires site information associated with the image ID of the endoscopic image 148a. Here, the additional information of the endoscopic image with image ID 3 includes site information "upper body, lesser curvature," and therefore, based on the presence information and site information, the first marking unit 102 recognizes that a biopsy site exists in the "upper body, lesser curvature," and places a first biopsy mark on the schematic diagram indicating that a biopsy has been performed or should be performed. When the second marking unit 104 places a second biopsy mark 170a on the endoscopic image 148a according to user operation, the first marking unit 102 may immediately place the first biopsy mark on the schematic diagram. When the screen shown in Figure 9 is displayed on the display device 12b, and the operation reception unit 82 receives a user operation to switch the displayed image to the schematic diagram, the display screen generation unit 100 displays the schematic diagram with the first biopsy mark placed on it on the display device 12b instead of the endoscopic image 148a.

[0046] Figure 10 shows an example of the schema editing screen. Based on the fact that the second biopsy mark 170a has been added to the endoscopic image 148a, the first marking unit 102 automatically adds a first biopsy mark 160a to the position corresponding to the biopsy site in the organ model image, indicating that a biopsy has been performed or should be performed. The second biopsy mark 170a and the first biopsy mark 160a may be the same mark "<1". The display screen generation unit 100 displays the organ model image to which the first biopsy mark 160a has been added on the schema editing screen. Here, the first marking unit 102 automatically adds the first biopsy mark 160a to the partial region (see Figure 5) corresponding to the "lesser curvature of the upper body". It is preferable that the first marking unit 102 places the first biopsy mark 160a approximately in the center of the partial region of the "lesser curvature of the upper body". Note that while the organ model image 162 shown in Figure 10 has dividing lines for each part, these lines are not required.

[0047] Figure 11 shows an example of an editing screen for another endoscopic image. When the user double-clicks the endoscopic image with image ID 4 on the selection screen (see Figure 7), the display screen generation unit 100 generates a display screen including an editable, enlarged endoscopic image 148b and displays it on the display device 12b. By enlarging the endoscopic image, the user can more easily confirm the biopsy site 150b included in the endoscopic image 148b. The biopsy site 150b may be lesion tissue photographed before the biopsy, or lesion tissue photographed after the biopsy.

[0048] When the user presses the biopsy designation button 152, the second marking unit 104 becomes ready to apply a second biopsy mark to the enlarged endoscopic image 148b in response to the user's operation, indicating that a biopsy has been performed or should be performed. When the operation reception unit 82 receives a user operation to place the second biopsy mark on the endoscopic image 148b, the second marking unit 104 applies the second biopsy mark 170b at the position specified by the user. In the example shown in Figure 11, the second biopsy mark 170b "<2" is applied near the biopsy site 150b.

[0049] At this time, the first marking unit 102 acquires presence information indicating the presence of a biopsy site in the endoscopic image 148b, and location information indicating the part included in the endoscopic image 148b. Here, the first marking unit 102 acquires location information "upper body lesser curvature" associated with image ID 4 of the endoscopic image 148b. Therefore, based on the presence information and location information, the first marking unit 102 recognizes that a biopsy site exists in the "upper body lesser curvature" and places the first biopsy mark at the position corresponding to the biopsy site in the schematic diagram.

[0050] Figure 12 shows an example of an editing screen for another endoscopic image. When the user double-clicks the endoscopic image with image ID 5 on the selection screen (see Figure 7), the display screen generation unit 100 generates a display screen including an editable, enlarged endoscopic image 148c and displays it on the display device 12b. When the operation reception unit 82 receives a user operation to place a second biopsy mark on the endoscopic image 148c, the second mark assignment unit 104 assigns the second biopsy mark 170c at the position specified by the user operation. In the example shown in Figure 12, the second biopsy mark 170c "<3" is assigned near the biopsy site 150c.

[0051] At this time, the first marking unit 102 acquires presence information indicating the presence of a biopsy site in the endoscopic image 148c, and location information indicating the part included in the endoscopic image 148c. Here, the first marking unit 102 acquires location information "upper body lesser curvature" associated with image ID 5 of the endoscopic image 148c. Therefore, based on the presence information and location information, the first marking unit 102 recognizes that a biopsy site exists in "upper body lesser curvature" and places the first biopsy mark at the position corresponding to the biopsy site in the schematic diagram.

[0052] When the screen shown in Figure 12 is displayed on the display device 12b, and the operation reception unit 82 receives a user operation to switch the displayed image to a schematic diagram, the display screen generation unit 100 displays a schematic diagram with multiple first biopsy marks instead of the endoscopic image 148c.

[0053] Figure 13 shows an example of the schema diagram editing screen. The first marking unit 102 automatically assigns first biopsy marks 160a, 160b, and 160c, corresponding to the second biopsy marks 170a, 170b, and 170c assigned to multiple endoscopic images, to positions corresponding to the biopsy sites in the organ model images. The first biopsy mark 160a may be the same as the second biopsy mark 170a, "<1", the first biopsy mark 160b may be the same as the second biopsy mark 170b, "<2", and the first biopsy mark 160c may be the same as the second biopsy mark 170c, "<3".

[0054] In this embodiment, the three endoscopic images with image IDs 3 to 5 are images taken of the same area (upper body, lesser curvature), and each contains a different biopsy site. When there are multiple endoscopic images taken of different biopsy sites in the same area, the first marking unit 102 determines the positions of multiple first biopsy marks 160a, 160b, and 160c in the partial region corresponding to the same area, based on the order in which the images were taken.

[0055] Specifically, the first marking unit 102 acquires timing information indicating the timing of the acquisition of the three endoscopic images with image IDs 3 to 5, and determines the acquisition order of the three endoscopic images. The timing information may be the frame number of the image or the time of acquisition, or it may simply be the serial number included in the image ID (assigned in the order of capture). In this example, it is determined that the endoscopic images were captured in the order of image ID 3, image ID 4, and image ID 5. The first marking unit 102 places the first biopsy marks 160a, 160b, and 160c in the partial region of the "upper body, lesser curvature" in accordance with the acquisition order, so as not to overlap.

[0056] The first marking unit 102 may estimate the direction of movement of the endoscope during the endoscopic examination based on the part information of each of the multiple captured endoscopic images and timing information indicating the timing of acquisition, and determine the positions of the first biopsy marks 160a, 160b, and 160c in the partial region of the "upper body, looking up at the lesser curvature" based on the estimated direction of movement. The first marking unit 102 estimates the direction of movement of the endoscope from endoscopic images that include other parts captured chronologically before and after the endoscopic images ID3 to 5, which are images of the biopsy site. Referring to Figure 7, the endoscopic image ID2, which was captured chronologically before, is an image of the "dome," and the endoscopic image ID6, which was captured chronologically after, is an image of the "mid body, looking down at the lesser curvature." Therefore, the first marking unit 102 recognizes that the endoscope has moved in the order of "dome" → "upper body lesser curvature" → "mid body lesser curvature downwards" and has photographed each part.

[0057] Figure 14 shows the estimated direction of movement of the endoscope. The first marking unit 102 determines the positions of the first biopsy marks 160a, 160b, and 160c in the "upper body lesser curvature" region according to the estimated direction of movement of the endoscope and the acquisition order of the three endoscope images with image IDs 3 to 5. Specifically, the first marking unit 102 places the first biopsy mark 160a near the entrance of the "upper body lesser curvature" region in the direction of endoscope movement, places the first biopsy mark 160c near the exit of the said region, and places the first biopsy mark 160b between the first biopsy marks 160a and 160c (see Figure 13). The first marking unit 102 arranges multiple first biopsy marks 160a, 160b, and 160c based on the endoscopic movement direction and imaging sequence, making it possible to approximate the positions of the first biopsy marks 160a, 160b, and 160c on the schematic diagram to the actual positions of the biopsy sites 150a, 150b, and 150c.

[0058] In the report creation process, the user selects images to attach to the report and enters the inspection results in the input area 58 on the report creation screen to create the report. When the user operates the registration button (see Figure 6), the registration processing unit 106 registers the contents entered on the report creation screen to the server device 2, and the report creation process is completed.

[0059] The present disclosure has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications are also within the scope of the present disclosure. In the embodiments, the endoscope observation device 5 transmits the user-captured images to the image storage device 8, but in modifications, the image analysis device 3 may transmit the user-captured images to the image storage device 8. Also, in the embodiments, the information processing device 11b has the processing unit 80, but in modifications, the server device 2 may have the processing unit 80.

[0060] In this embodiment, the display screen generation unit 100 displays the organ model image with the first biopsy mark and the endoscopic image with the second biopsy mark on different screens. However, in a modified example, the organ model image with the first biopsy mark and the endoscopic image with the second biopsy mark may be displayed simultaneously on the same screen.

[0061] Figure 15 shows an example of an editing screen for an endoscopic image including an organ model image 162. In this example, the display screen generation unit 100 displays the endoscopic image 148c with image ID 5 and the organ model image 162 on the same screen. The operation reception unit 82 receives a user operation to place a second biopsy mark on the endoscopic image 148c, and the second mark assignment unit 104 assigns the second biopsy mark 170c at the position specified by the user operation. Then, the first mark assignment unit 102 assigns the first biopsy mark 160c, corresponding to the second biopsy mark 170c, to the organ model image 162. At this time, the first mark assignment unit 102 may determine the placement positions of multiple first biopsy marks 160a, 160b, and 160c so that they do not overlap, based on the estimated direction of movement of the endoscope and the timing of imaging of multiple biopsy sites in the same area. By displaying the endoscopic image 148c and the organ model image 162 on the same screen, users can easily confirm that when a second biopsy mark 170c is added to the endoscopic image 148c, a first biopsy mark 160c corresponding to the second biopsy mark 170c is automatically added to the organ model image 162.

[0062] In this embodiment, the operation reception unit 82 receives a user operation to place the second biopsy mark on the endoscopic image 148c, and the second mark application unit 104 then applies the second biopsy mark to the endoscopic image. In a modified example, the second mark application unit 104 may apply the second biopsy mark based on image analysis information output by the image analysis device 3.

[0063] In this modified version, the image analysis device 3 has the function of detecting biopsy sites included in endoscopic images. The image analysis device 3 confirms whether or not a biopsy site is included in the endoscopic image by image analysis. For each endoscopic image, the image analysis device 3 outputs information indicating whether or not a biopsy site is included, and if a biopsy site is included, it outputs location information indicating the location of the biopsy site. In other words, when the image analysis device 3 recognizes that a biopsy site is included in the endoscopic image, it outputs existence information indicating the presence of a biopsy site in the endoscopic image, and location information indicating the location of the biopsy site (location coordinates within the endoscopic image). In this modified version, this existence information and location information are provided to the server device 2 as part of the image analysis information.

[0064] In this modified version, the second marking unit 104 adds a second biopsy mark to the endoscopic image based on the presence information and position information output by the image analysis device 3, and simultaneously, the first marking unit 102 adds a first biopsy mark to the organ model image. According to this modified version, the addition of the second biopsy mark to the endoscopic image and the addition of the first biopsy mark to the organ model image can all be performed automatically, further supporting the efficiency of the user's report creation work. [Industrial applicability]

[0065] This disclosure can be used in technical fields that support the preparation of reports. [Explanation of Symbols]

[0066] 1...Medical support system, 2...Server device, 3...Image analysis device, 4...Network, 5...Endoscope observation device, 6...Display device, 7...Endoscope, 8...Image storage device, 9...Endoscope system, 10a,10b...Terminal device, 11a,11b...Information processing device, 12a,12b...Display device, 20...Communication unit, 30...Processing unit, 40...Order information acquisition unit, 42...Additional information acquisition unit, 50...List display area, 54a...Recorded image tab, 54b...Report tab, 56...Attachment Image display area, 58...Input area, 60...Storage device, 62...Order information storage unit, 64...Additional information storage unit, 76...Communication unit, 78...Input unit, 80...Processing unit, 82...Operation reception unit, 84...Acquisition unit, 86...Image acquisition unit, 88...Additional information acquisition unit, 100...Display screen generation unit, 102...First mark assignment unit, 104...Second mark assignment unit, 106...Registration processing unit, 120...Storage device, 122...Image storage unit, 124...Additional information storage unit, 126...Schema image storage unit.

Claims

1. A medical support system comprising one or more processors having hardware, wherein the organ model image includes multiple subregions corresponding to multiple parts of an organ, The one or more processors described above are: Based on location information indicating the region contained in each of the multiple endoscopic images taken during a single endoscopic examination, and timing information indicating the timing of each of the multiple endoscopic images being taken, the direction of movement of the endoscope during the endoscopic examination is estimated. For each endoscopic image, information indicating the presence of a biopsy site, location information, and timing information of the endoscopic image are acquired. Based on the existence information and the site information, a first mark indicating that a biopsy has been performed or should be performed is applied to the partial region in the organ model image corresponding to the site indicated by the site information. When there are multiple endoscopic images taken of different biopsy sites in the same area, the positions of multiple first marks in the partial region corresponding to the same area are determined based on the estimated direction of movement of the endoscope and the order in which the multiple endoscopic images taken of different biopsy sites in the same area were taken. Display the organ model image to which multiple first marks have been assigned. A medical support system characterized by the following features.

2. The one or more processors described above are: A second mark is added to the endoscopic image of the biopsy site to indicate that a biopsy has been performed or should be performed. The medical support system according to feature 1.

3. The one or more processors described above are: The system accepts a user operation to place the second mark on the endoscope image. In accordance with the user operation described above, the second mark is added to the endoscopic image. The medical support system according to feature 2.

4. The one or more processors described above are: Based on the endoscopic image, the presence information and location information indicating the location of the biopsy site are obtained. Based on the existence information and the position information, the second mark is added to the endoscopic image. The medical support system according to feature 2.

5. The one or more processors described above are: Based on the endoscopic image, the region information is acquired. The medical support system according to feature 1.

6. The one or more processors described above are: The endoscopic image is input into the trained model, and the region information output from the trained model is obtained. The aforementioned trained model is generated by machine learning using training endoscopic images and information indicating the body parts contained in the endoscopic images as training data. The medical support system according to feature 5.

7. The one or more processors described above are: The organ model image to which the first mark is assigned and the endoscopic image to which the second mark is assigned are displayed simultaneously. The medical support system according to feature 2.

8. The one or more processors described above are: The endoscopic image to which the second mark has been assigned is displayed, When a user operation to switch the displayed image is received, the organ model image to which the first mark is assigned is displayed instead of the endoscopic image. The medical support system according to feature 2.

9. The aforementioned existence information includes information indicating the presence of a site to be biopsied, or information indicating the presence of a site that has already been biopsied. The medical support system according to feature 1.

10. The one or more processors described above are: In the aforementioned partial region, the positions of the multiple first marks are determined such that the multiple first marks do not overlap. The medical support system according to feature 1.

11. A medical support method wherein the organ model image includes multiple subregions corresponding to multiple parts of the organ, Based on location information indicating the region contained in each of the multiple endoscopic images taken during a single endoscopic examination, and timing information indicating the timing of each of the multiple endoscopic images being taken, the direction of movement of the endoscope during the endoscopic examination is estimated. For each endoscopic image, information indicating the presence of a biopsy site, location information, and timing information of the endoscopic image are acquired. Based on the existence information and the site information, a first mark indicating that a biopsy has been performed or should be performed is applied to the partial region in the organ model image corresponding to the site indicated by the site information. When there are multiple endoscopic images taken of different biopsy sites in the same area, the positions of multiple first marks in the partial region corresponding to the same area are determined based on the estimated direction of movement of the endoscope and the order in which the multiple endoscopic images taken of different biopsy sites in the same area were taken. Display the organ model image to which multiple first marks have been assigned. A medical support method characterized by the following features.

12. A recording medium on which a program is stored, wherein the organ model image includes multiple subregions corresponding to multiple parts of an organ, On the computer, A function to estimate the direction of movement of the endoscope during an endoscopic examination based on location information indicating the region contained in each of the multiple endoscopic images taken during a single endoscopic examination, and timing information indicating the timing of acquisition of each of the multiple endoscopic images. A function to acquire presence information indicating the presence of a biopsy site in the endoscopic image, and the location information and timing information of the endoscopic image for each endoscopic image, A function to assign a first mark indicating that a biopsy has been performed or should be performed, based on the existence information and the site information, to the partial region in the organ model image corresponding to the site indicated by the site information, When there are multiple endoscopic images taken of different biopsy sites in the same area, the function determines the positions of multiple first marks in the partial region corresponding to the same area based on the estimated direction of movement of the endoscope and the order in which the multiple endoscopic images taken of different biopsy sites in the same area were taken. A recording medium that stores a program for realizing a function to display the organ model images to which multiple first marks have been assigned.

Images