Medical Image Processing System

The medical image processing system synchronizes and displays endoscopic and ultrasound images to efficiently observe and analyze swallowing tests, reducing patient burden and improving test efficiency.

JP7579750B2Active Publication Date: 2024-11-08FUJIFILM CORP
View PDF 14 Cites 0 Cited by

Patent Information

Application Number
JP2021086548
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-05-21
Publication Date
2024-11-08
Estimated Expiration
2041-05-21

AI Technical Summary

Technical Problem

Existing swallowing tests using multiple medical imaging systems are burdensome for patients and inefficient for healthcare providers due to time lags in video playback and complex result management.

Method used

A medical image processing system that synchronizes and displays endoscopic and ultrasound images in real-time, allowing for simultaneous observation of swallowing stages, and includes a processor to tag and extract index videos for efficient analysis.

Benefits of technology

Reduces patient burden and enhances the efficiency and accuracy of swallowing tests by integrating and synchronizing multiple imaging systems, facilitating detailed observation and analysis of swallowing dynamics.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579750000001
    Figure 0007579750000001
  • Figure 0007579750000002
    Figure 0007579750000002
  • Figure 0007579750000003
    Figure 0007579750000003
Patent Text Reader

Abstract

To provide a medical image processing system which reduces a burden on a patient and allows a medical doctor to efficiently observe deglutition in deglutition check with a plurality of check systems.SOLUTION: Deglutition checks using a plurality of portable deglutition check systems are performed simultaneously to reduce a burden on a patient. The system acquires an endoscopic image signal obtained through deglutition check of a laryngopharynx part using an endoscope inserted from a patient's nasal cavity and an ultrasonic image signal obtained by observing the air duct and the esophagus, observing the laryngopharynx part, and observing the tongue and the vicinity of the hyoid bone with an ultrasonic check terminal abutted to the patient's neck, performs synchronous processing on the basis of an imaging period, and simultaneously displays them on a display 14.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to medical image processing systems. [Background technology]

[0002] When an endoscope is inserted into a subject and observation is performed, images are mainly taken of the pharynx in a swallowing test. In an endoscopic swallowing test, the tip of the endoscope inserted through the nasal cavity takes an image of the pharynx and observes the swallowing during the pharyngeal phase. In swallowing, there is the oral phase in which food is sent from the tongue to the pharynx, the pharyngeal phase in which food is sent from the pharynx to the esophagus, and the esophageal phase in which food is sent through the esophagus to the stomach. The swallowing ability of a patient can be examined by observing the pharyngeal phase with an endoscope, but swallowing can be evaluated and judged in more detail if information on swallowing during the oral or esophageal phase can be obtained in addition to the pharyngeal phase swallowing obtained by photographing the pharynx. In addition, swallowing can be understood in more detail by examining swallowing during the pharyngeal phase with a different method. A swallowing contrast test uses swallowed food with added contrast agent and observes its dynamics with an X-ray fluoroscopy device, and observes the morphology and function of parts related to swallowing, aspiration, and residual swallowed food. In recent years, observations using ultrasound examination devices have also been performed. Ultrasound devices do not pose the risk of radiation exposure like X-rays, can be used on the body surface, and are an examination method that places less strain on the patient. Ultrasound examinations include examinations to obtain information on esophageal swallowing by observing the esophagus and airway, examinations to observe the swallowing, aspiration, and residue of food and drink by observing the laryngopharynx, and examinations to evaluate swallowing movements by observing the morphology of the tongue, hyoid bone area, and laryngeal muscles.

[0003] However, because multiple testing systems are used, multiple swallowing tests in a short period of time place a burden on the patient. Also, it is inefficient for the examiners to check the results of each swallowing test separately, and management of the test results tends to be cumbersome. In addition, reviewing the results takes time.

[0004] Patent Document 1 describes performing a swallowing test using an endoscope and evaluating swallowing function using an X-ray fluoroscopy device, and displaying the acquired endoscopic video and X-ray contrast image on the same screen. Patent Document 2 describes taking images using medical devices such as an endoscope, a radiography device, and an ultrasound diagnostic device when performing a specific diagnosis or treatment, and wirelessly transmitting each of the acquired images to the same tablet PC. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Utility Model Registration No. 3209885 [Patent Document 2] Patent No. 5823535 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, the simultaneous display of endoscopic video and X-ray video is in real time, so there may be a discrepancy in the playback timing. In addition, there is no description of the specific swallowing observation site or the implementation of an ultrasound examination. In Patent Document 2, diagnosis and treatment are performed in an operating room using multiple medical devices, but there is no description of swallowing examination, nor is there any description of the imaging device being portable. In light of the above, it is desirable to accurately and efficiently observe swallowing by simultaneously playing back videos of swallowing captured by multiple systems without a time lag.

[0007] An object of the present invention is to provide a medical image processing system that reduces the burden on patients and enables doctors to efficiently observe swallowing in swallowing examinations using multiple examination systems. [Means for solving the problem]

[0008] The medical image processing system of the present invention comprises a processor that receives an endoscopic video signal obtained by imaging a swallowing test of the patient's throat using an endoscope inserted through the patient's nasal cavity, receives ultrasound video signals of the esophagus, airway, laryngopharynx, tongue or area around the hyoid bone obtained by an ultrasound examination terminal placed on the patient's neck, and simultaneously displays the endoscopic video signal and the ultrasound video signal on a display.

[0009] It is preferable that, when acquiring the endoscopic video signal and the ultrasound video signal, the processor records the shooting time based on the processor's time, performs synchronization processing to synchronize the endoscopic video signal and the ultrasound video signal based on the shooting time, and plays the synchronized endoscopic video signal and the ultrasound video signal in synchronization on the display.

[0010] It is preferable that the endoscopic video signal and the ultrasound video signal are reproduced automatically or by user operation.

[0011] It is preferable that, when acquiring the endoscopic video signal and the ultrasound video signal, the processor records the shooting time based on the processor's time, performs synchronization processing of the endoscopic video signal and the ultrasound video signal based on the shooting time, detects the swallowing timing from the endoscopic video signal, tags the swallowing timing, extracts the endoscopic index video, tags the ultrasound video signal that has the same shooting time as the endoscopic video signal at the swallowing timing, extracts the ultrasound index video, and plays the endoscopic index video and the ultrasound index video on the display.

[0012] It is preferable that, when acquiring the endoscopic video signal and the ultrasound video signal, the processor records the shooting time based on the processor's time, performs synchronization processing of the endoscopic video signal and the ultrasound video signal based on the shooting time, detects swallowing timing from the endoscopic video signal, detects non-swallowing timing from the ultrasound video signal, tags the endoscopic video signal at the swallowing timing that has the same shooting time as the ultrasound video signal other than the non-swallowing timing, extracts the endoscopic index video, tags the ultrasound video signal at the swallowing timing that has the same shooting time as the endoscopic video signal at the swallowing timing, extracts the ultrasound index video, and plays the endoscopic index video and the ultrasound index video on the display.

[0013] It is preferable that the endoscopic index video and the ultrasound index video are played automatically or by user operation.

[0014] It is preferable that the processor detects swallowing timing as either normal swallowing, erroneous swallowing, or unknown swallowing from the endoscopic video signal and the ultrasound video signal, and tags the frame image in which the swallowing timing is detected as swallowing timing detection.

[0015] The endoscopic video signal and the ultrasound video signal are composed of a series of frame images, and the processor determines whether or not there is a swallowing timing in the endoscopic video signal by calculating the amount of blur between the frame images, calculating key points based on the frame images, or using the difference in pixel values ​​between the frame images, and it is preferable to use machine learning to determine whether or not there is a swallowing timing in the ultrasound video signal.

[0016] The processor preferably analyses at least one of the endoscopic index video or the ultrasound index video to identify the type of swallowing test.

[0017] It is preferable that the processor classifies the type of swallowing test according to the type of swallowing test food and the swallowing result, and assigns index numbers to the endoscopic index video and the ultrasound index video based on the classification.

[0018] In acquiring the endoscopic video signal and the ultrasonic video signal, the processor preferably performs imaging at a frame rate of 60 fps, and counts the imaging time in units of 1 / 100th of a second.

[0019] The processor preferably simultaneously displays the endoscopic video signal and the ultrasound video signal in real time on the display without synchronization.

[0020] The processor may receive a radiographic video signal obtained by observing a swallowing examination of a patient using a radiation observation device instead of observation using an ultrasound examination terminal, and display the endoscopic video signal and the radiographic video signal on a display.

[0021] In addition to observation using the ultrasound examination terminal, the processor may receive a radiographic image signal obtained by observing a swallowing examination of a patient using a radiation observation device, and display the endoscopic image signal, the ultrasound image signal, and the radiographic image signal on the display. Effect of the Invention

[0022] By performing swallowing tests using multiple swallowing test systems simultaneously, the burden on patients is reduced, and by displaying the acquired videos on the same screen, the burden on doctors is reduced, allowing for efficient and accurate swallowing observations. [Brief description of the drawings]

[0023] [Figure 1] 1 is a schematic diagram of a medical imaging system having an endoscope system and an ultrasound system. [Diagram 2] FIG. 1 illustrates the stages of swallowing food. [Diagram 3] FIG. 2 is a diagram showing an observation site observed by an endoscope and an ultrasound examination terminal in a swallowing examination. [Figure 4]FIG. 2 is a block diagram showing the functions of the processor device; [Diagram 5] FIG. 13 is a diagram showing endoscopic video and ultrasound video being synchronized and displayed simultaneously on a screen. [Figure 6] FIG. 13 is a diagram showing an endoscopic video displayed alone on a screen. [Figure 7] 1A-1D are images of the pharynx with and without swallowing food. [Figure 8] 1A-1D are images of the esophagus with and without swallowing food. [Figure 9] FIG. 13 is a diagram showing extraction of an index video from an endoscopic video and an ultrasound video. [Figure 10] FIG. 13 is a diagram showing the classification results of swallowing and allocation to index numbers. [Figure 11] FIG. 13 is an image diagram showing an endoscopic index video and an ultrasound index video being synchronously played back on the screen at the same time. [Figure 12] FIG. 1 is an image diagram showing an endoscopic index video being played back and displayed. [Figure 13] 13 is a flowchart showing synchronous playback of endoscopic videos and ultrasound videos. [Figure 14] FIG. 11 is a schematic diagram of a medical image processing system having an endoscope system and a radiation system in a second embodiment. [Figure 15] FIG. 13 is an image diagram showing synchronous playback of an endoscopic video and a radiological video. [Figure 16] 13 is an image diagram showing an index video extracted from an endoscopic video and a radiological video. FIG. [Figure 17] FIG. 11 is a schematic diagram of a medical image processing system having an endoscope system, an ultrasound system, and a radiation system in a third embodiment. [Figure 18] FIG. 13 is an image diagram showing synchronous playback of an endoscopic video, an ultrasound video, and a radiological video. [Figure 19] 11 is an image diagram showing an index video extracted from an endoscopic video, an ultrasound video, and a radiological video. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] [First embodiment] 1, a medical image processing system 10 includes an endoscope system 11, an ultrasound system 12, a processor device 13, a display 14, and a user interface 15. The processor device 13 is electrically connected to the endoscope system 11, the ultrasound system 12, the display 14, and the user interface 15. The processor device 13 acquires video signals acquired in a swallowing test from an endoscope 11a constituting the endoscope system 11 and an ultrasound examination terminal 12a constituting the ultrasound system 12. The user interface 15 is an input device for inputting settings to the processor device 13, and includes a keyboard, etc.

[0025] The endoscope 11a and the ultrasound examination terminal 12a are battery operated to acquire images of the swallowing examination of the patient. Since swallowing is a dynamic process, a video signal that becomes video data is acquired in the swallowing examination. It is preferable that the video signal is wirelessly transmitted to the processor device 13. If there is no particular designation for the swallowing imaging, white light is used as the illumination light, a video signal of 60 frames per second (60 fps (frames per second)) is acquired, and the imaging time based on the processor device 13 is recorded. If the video signal is 60 fps, it is preferable that the time is counted in units of 1 / 100th of a second. It is preferable that the processor device 13, the display 14, and the user interface 15 are integrated as a tablet terminal, and the endoscope system 11 and the ultrasound system 12 are combined together to form a portable medical image processing system.

[0026] As shown in FIG. 2, swallowing is a series of movements in which food F put in the mouth is chewed, swallowed, and sent to the stomach. The progression of swallowing includes an "oral phase" in which food F put in the mouth is sent from the tongue to the pharynx, a "pharyngeal phase" in which food F is sent from the pharynx to the esophagus, and an "esophageal phase" in which food F is sent from the esophagus to the stomach. In this embodiment, the pharynx is inspected with the endoscope 11a to obtain a video of the "pharyngeal phase" of swallowing, and the esophagus is inspected with the ultrasound inspection terminal 12a to obtain a video of the "esophageal phase" of swallowing. The food F is a plurality of liquids or solids that are harmless to the human body when swallowed and have different ease of swallowing. For example, milk, a colored aqueous solution, and pudding are used.

[0027] In a swallowing test, the patient holds food F in the mouth, swallows it at the pharynx, and photographs the food as it passes through the esophagus and into the stomach. Even if the patient is unable to swallow food F, the test should not be stopped and photographs should be taken. When performing a swallowing test, it is preferable to photograph not one swallow, but multiple swallows, as far as the patient can handle.

[0028] As shown in FIG. 3, the endoscope 11a is a swallowing endoscope that is inserted from the patient's nasal cavity Nc and photographs the swallowing of food F around the pharynx illuminated with illumination light with the endoscope tip 11b at the position R. The ultrasound examination terminal 12a is placed on the patient's neck and the ultrasound examination terminal contact part 12b at the tip is placed in contact with the throat Th to photograph the swallowing of food F from the airway to the esophagus. In addition, it is preferable to apply a dedicated gel to the part of the patient's body surface that the ultrasound examination terminal contact part 12b contacts beforehand. The endoscope tip part 11b photographs approximately horizontally to the ground, and the ultrasound examination terminal contact part 12b photographs approximately vertically to the ground. The ultrasound examination terminal 12a can also photograph the swallowing in the pharyngeal and oral phases in addition to the esophageal phase by changing the part to which it is placed.

[0029] 4, in the processor device 13, programs related to processing such as image processing are stored in a program memory (not shown). In the processor device 13, a central control unit 20 constituted by a processor operates a program in the program memory to realize the functions of an input receiving unit 21, a display control unit 22, a storage memory 23, an endoscopic moving image processing unit 24, an ultrasonic moving image processing unit 25, and a moving image editing unit 30. In addition, with the realization of the function of the moving image editing unit 30, the functions of a temporary storage memory 31, a moving image synchronization unit 32, a swallowing timing detection unit 33, an index moving image extraction unit 34, and a swallowing classification unit 35 are realized. The function of the swallowing timing detection unit 33 may be included in the endoscope system 11.

[0030] An endoscopic video signal acquired by the endoscopic system 11 is sent to the video editing unit 30 via an endoscopic video processing unit 24, and an ultrasound video signal acquired by the ultrasound system 12 is sent to the video editing unit 30 via an ultrasound video processing unit 25. The input receiving unit 21 is connected to the user interface 15. The display control unit 22 controls the display 14 to display the video synchronized by the video synchronization unit 32. The storage memory 23 temporarily stores each video signal as a video.

[0031] After the swallowing test is completed, if a video is being recorded, the mode can be switched to either normal video synchronization mode or index video synchronization mode. The normal video synchronization mode stores and synchronizes the video, while the index video synchronization mode stores and synchronizes the video, extracts the index video, and classifies the type of swallowing test. During the swallowing test, the mode is set to real-time display mode, which displays the captured content in real time.

[0032] When the swallowing test is completed, the real-time display mode is switched to either the normal video synchronization mode or the index video synchronization mode. It is preferable to set which mode to switch to via the user interface 15 before the real-time display mode ends. Note that the video is acquired after the swallowing test is completed, but each video signal may be processed in real time during the swallowing test.

[0033] The normal video synchronization mode will now be described. As shown in Fig. 5, each acquired video signal is temporarily stored in the temporary storage memory 31 as an endoscopic video 41 and an ultrasound video 51, and a synchronization process is performed in the video synchronization unit 32 based on the shooting time. After synchronization, the endoscopic video 41 and the ultrasound video 51 are arranged on the display 14 via the display control unit 22 and are played back in synchronization. In addition, a play button 61, a fast-rewind button 62, a fast-forward button 63, a pause button 64, a seek bar 65, a slider 66, a repeat play button 67, and a screen switching button 68 are provided, and the user can use these functions via the user interface 15.

[0034] In the temporary storage memory 31, each video signal recording the swallowing test sent to the video editing unit 30 is temporarily stored as an endoscopic video 41 and an ultrasound video 51. Each video that has been temporarily stored is transmitted to the video synchronization unit 32.

[0035] The video synchronization unit 32 performs a synchronization process of the endoscopic video 41 and the ultrasound video 51 temporarily stored in the temporary storage memory 31. In the synchronization process, the endoscopic video 41 and the ultrasound video 51 are synchronized on a frame-by-frame basis based on the shooting time, enabling simultaneous playback on the playback screen. In addition, synchronization is also performed for information such as patient information and shooting date and time. Note that the shooting time used as the basis for synchronization is a time that the endoscope system 11 and the ultrasound system 12 have previously synchronized with the processor device 13, and the endoscopic video 41 and the ultrasound video 51 after the synchronization process are sent to the display control unit 22.

[0036] The display control unit 22 displays the endoscopic video 41 and the ultrasound video 51 that have been synchronized on the playback screen of the display 14, and playback is performed automatically or by user operation. The user observes the state of simultaneous playback. On the playback screen, the endoscopic video 41 and the ultrasound video 51 that have finished automatic playback can be repeatedly played back by pressing the playback button 61, and the user can go back to a missed scene with the fast rewind button 62, increase the video playback speed with the fast forward button 63, and stop playback at any timing with the pause button 64. The playback status of the video is represented by the position of a slider 66 on a seek bar 65, and the user can freely change the playback point by moving the position of the slider 66 by a user operation such as dragging. When the repeat playback button 67 is selected, the endoscopic video 41 and the ultrasound video 51 that have finished playback are repeatedly played back.

[0037] 6, the video playback on the display 14 can be switched from a dual-screen synchronous display of the synchronized endoscopic video 41 and ultrasound video 51 to a single-screen display by using the screen switching button 68. It is preferable that the endoscopic video 41 is displayed on a single screen, and when the screen switching button 68 is selected again, the screen is switched to a single-screen display of the ultrasound video 51, and when the screen switching button 68 is further selected again, the screen is switched to the dual-screen synchronous display. In the case of a single-screen display, a video information display field 69 is expanded to display video information such as patient information, and the user may input the video information via the user interface 15.

[0038] The user can confirm, add, or edit the information of the endoscopic video 41 and the ultrasound video 51 displayed in the video information display field 69 from the user interface 15 via the input receiving unit 21. The information is, for example, findings obtained by playing back the video, and characteristics such as the patient's gender and age group. The endoscopic video 41 and the ultrasound video 51 including the edited video information are stored in the storage memory 23.

[0039] The index video synchronization mode will be described. The endoscopic video 41 and the ultrasound video 51 are temporarily stored in the temporary storage memory 31, as in the normal video synchronization mode, and the video synchronization unit 32 performs synchronization processing based on the shooting time. After synchronization, the swallowing timing detection unit 33 detects and tags frame images having swallowing timing from the endoscopic video 41 and the ultrasound video 51. In order to efficiently observe swallowing, the index video extraction unit 34 extracts index videos as scenes where swallowing has occurred. The swallowing classification unit 35 automatically classifies the type of index video, etc. After automatic classification, the images are arranged on the display 14 via the display control unit 22 and played back in synchronization.

[0040] In both the normal video synchronization mode and the index video synchronization mode, temporary storage in the temporary storage memory 31 and synchronization processing by the video synchronization unit 32 are performed in common. Therefore, switching between the normal video synchronization mode and the index video synchronization mode may be performed by the time the synchronization processing is completed.

[0041] 7 shows an endoscopic swallowing frame image 42 showing food F passing through the pharynx and an endoscopic non-swallowing frame image 43 showing food F not passing through the pharynx, which are part of an endoscopic video 41. When filming the endoscopic video 41, filming begins before or after the patient puts food F in his or her mouth and ends after swallowing, including aspiration, has occurred, but because the swallowing is captured for a few seconds at most, the frame images that make up the endoscopic video 41 are mostly endoscopic non-swallowing frame images 43.

[0042] 8 shows an ultrasound swallowing frame image 52 showing food F passing through the esophagus and an ultrasound non-swallowing frame image 53 showing food F not passing through the esophagus, which constitute an ultrasound video 51. Since the shooting of the ultrasound video 51 starts and ends in the same manner as the endoscopic video 41, the ultrasound video 51 is made up of many ultrasound non-swallowing frame images 53 that do not show food F.

[0043] The swallowing timing detection unit 33 performs a pharyngeal stage swallowing detection process in which the endoscopic video 41 is image-analyzed using a tool dedicated to machine learning such as deep learning, and determines whether or not it has a pharyngeal stage swallowing timing, which is the dynamic state when food F passes through the pharynx. The portion determined to be a pharyngeal stage swallowing timing is the endoscopic swallowing frame image 42 in which food F is shown, and when it is detected as a swallowing timing by the pharyngeal stage swallowing detection process, it is tagged as "swallowing timing detected."

[0044] Similarly to the endoscopic video 41, the ultrasound video 51 is subjected to image analysis using a dedicated machine learning tool such as deep learning, and an esophageal phase swallowing detection process is performed to determine whether or not the ultrasound video 51 has esophageal phase swallowing timing, which is the dynamic state when food F passes through the esophagus. The portion determined to have esophageal phase swallowing timing is the ultrasound swallowing frame image 52 showing food F, and if it is detected as swallowing timing by the esophageal phase swallowing detection process, it is tagged as "swallowing timing detected."

[0045] The tagged endoscopic video 41 and ultrasound video 51 are transmitted to the index video extraction unit 34. Note that the endoscopic video 41 and ultrasound video 51 in which the swallowing timing is not detected are preferably stored in the storage memory 23. At that time, they are deleted from the temporary storage memory 31.

[0046] The index video extraction unit 34 extracts an endoscopic index video 44, which is a group of consecutive frame images including a frame image tagged as "swallowing timing detection", from the endoscopic video 41, and similarly extracts an ultrasound index video 54, which is a group of consecutive frame images including a frame image tagged as "swallowing timing detection", from the ultrasound video 51.

[0047] Specifically, as shown in FIG. 9, a group of consecutive frame images of the esophageal phase swallowing frame image group 55 and a group of consecutive frame images for a certain period before and after the group of consecutive frame images are extracted from the pharyngeal phase swallowing frame image group 45 tagged with "swallowing timing detection" included in the endoscopic video 41 and the ultrasound video 51. The certain period is a time of about 3 or 5 seconds set in advance by the user, and it is preferable that the movement of the pharynx before the passage of food F or the movement of the esophagus after the passage of food F is captured. The arrows indicate the progress of time, and when the video is played back, it is played back and displayed in the direction of the arrow. The endoscopic index video 44 and the ultrasound index video 54 include the swallowing timing of the pharyngeal phase and the esophageal phase. The extracted endoscopic index video 44 and the ultrasound index video 54 are transmitted to the swallowing classification unit 35.

[0048] Since the endoscopic video 41 and the ultrasound video 51 are synchronized, the tagging result of one video may be synchronized with the other video to extract an index video. This makes it possible to check how the image in one video appears with respect to the swallowing timing detected only in the other video.

[0049] The swallowing timing captured by the endoscopic video 41 is the pharyngeal phase, and the swallowing timing captured by the ultrasound video 51 is the esophageal phase, so the periods during which food F is captured do not match when played back synchronously. For this reason, it is preferable that the range of extraction of each index video includes both the pharyngeal phase swallowing timing captured by the endoscopic video 41 and the esophageal phase swallowing timing captured by the ultrasound video 51. Note that swallowing during the pharyngeal phase and the esophageal phase is continuous, and the time difference between each swallowing timing is less than one second, so that the pharyngeal phase swallowing timing detected with a time difference of less than one second and the esophageal phase swallowing timing can be determined to be the same swallowing.

[0050] There are multiple patterns for detecting swallowing timing and tagging in order to extract each index video, and a first pattern is a pattern in which swallowing timing is detected and tagged in each of the above-described endoscopic video 41 and ultrasound video 51, and examples of patterns 2 to 6 are given below. When tagging is performed using synchronization processing, swallowing timing may be detected from each video or image signal without distinguishing between esophageal phase swallowing timing and pharyngeal phase swallowing timing.

[0051] In the second pattern, the swallowing timing detected in the endoscopic video 41 is tagged, and the swallowing timing is reflected in the portion of the synchronized ultrasound video 51 that is captured at the same time, and tagged as the swallowing timing of the ultrasound video 51. Therefore, the swallowing timing in the ultrasound video 51 is not detected.

[0052] In the third pattern, the swallowing timing detected in the ultrasound video 51 is tagged, and the swallowing timing is reflected in the portion of the synchronized endoscopic video 41 that is shot at the same time, and tagged as the swallowing timing of the endoscopic video 41. Therefore, the swallowing timing in the endoscopic video 41 is not detected.

[0053] The swallowing timing may be detected by combining both the endoscopic video 41 and the ultrasound video 51. When a single video is used to detect the swallowing timing, the timing may not be detected or may be erroneously detected. However, by using both the endoscopic video 41 and the ultrasound video 51 in combination, the accuracy of detection is improved. For example, in the endoscopic video 41, when the pharynx moves violently due to coughing or choking, the endoscopic video 41 cannot distinguish this from the behavior of swallowing, so it is erroneously detected as a swallowing timing, but in the ultrasound video 51, the muscle behavior is different, so it can be determined as non-swallowing, and it can be excluded from the swallowing timing detected in the endoscopic video 41. For example, in the case where the ultrasound video 51 is used to perform a test that uses a test meal and a case where the patient unconsciously swallows, it is detected as a swallowing timing in the ultrasound video 51, but the endoscopic video 41 can distinguish whether the test meal flows in or out, so that it is possible to exclude anything other than the swallowing of the test meal.

[0054] In the fourth pattern, after swallowing timing is detected in the endoscopic video 41 and the ultrasound video 51, the synchronized endoscopic video 41 and the ultrasound video 51 are compared, and tagging is performed only on the image signals with swallowing timings that match in shooting time.

[0055] In the fifth pattern, tagging is performed for the swallowing timing in the endoscopic video 41 and for timings other than non-swallowing timing in the synchronized ultrasound video 51. For example, an endoscopic video signal of the swallowing timing is detected from the endoscopic video 41, and an ultrasound video signal of at least the non-swallowing timing is detected from the ultrasound video 51 among the swallowing timing, non-swallowing timing, and unknown timing. Among the endoscopic video signals of the swallowing timing, tagging is performed for endoscopic video signals having the same shooting time as ultrasound video signals other than the non-swallowing timing, and among the ultrasound video signals other than the non-swallowing timing, tagging is performed for ultrasound video signals having the same shooting time as the endoscopic video signal of the swallowing timing.

[0056] In the sixth pattern, tagging is performed for swallowing timing in the ultrasound video 51 and non-swallowing timing in the synchronized endoscopic video 41. For example, an ultrasound video signal of swallowing timing is detected from the ultrasound video 51, and an endoscopic video signal of at least non-swallowing timing is detected from the endoscopic video 41 among swallowing timing, non-swallowing timing, and unknown timing. Among the ultrasound video signals of swallowing timing, tagging is performed for ultrasound video signals having the same shooting time as endoscopic video signals other than non-swallowing timing, and among the endoscopic video signals other than non-swallowing timing, tagging is performed for endoscopic video signals having the same shooting time as ultrasound video signals of swallowing timing.

[0057] The swallowing classification unit 35 classifies the type of swallowed food and the result of the swallowing test by analyzing the endoscopic index video 44 and the ultrasound index video 54, or assigns an index number. The swallowed food is the type of swallowed food F, such as swallowing saliva, swallowing milk, swallowing a colored aqueous solution, eating pudding, etc., and classifies the results as to whether the swallowing was normal, whether there was aspiration, or whether it was unknown. A score evaluation that assigns points according to the degree of aspiration may be used. These are automatically classified based on data learned by machine learning. It is preferable that the automatically classified information on the type of swallowing test food is assigned to the endoscopic index video 44 and the ultrasound index video 54. The classified endoscopic index video 44 and the ultrasound index video 54 are transmitted to the display control unit 22. The swallowing test result may be determined by the swallowing timing detection unit 33 and the result may be used.

[0058] The index number is a number used when a user searches for each index video stored in the storage memory 23, etc., and is preferably a unique character string with a small number of digits that indicates the order in which the swallowing timing was detected and the type of swallowing test food, etc., in alphanumeric characters. In this case, the types of swallowing test food are designated as saliva (sa), milk (mi), colored water (cw), pudding (pu), and unknown (un), and are further classified as normal swallowing (S), aspiration (A), or unknown swallowing (U). In addition, depending on the type of video, the endoscopic index video 44 is suffixed with (1), and the ultrasound index video 54 is suffixed with (2).

[0059] 10, for example, the swallowing type is identified by the swallowing classification unit 35 for consecutively extracted endoscopic index videos 44a, 44b, 44c, 44d, and 44e and the ultrasound index videos 54a, 54b, 54c, 54d, and 54e synchronized therewith. The result of one swallowing test is obtained from the synchronized endoscopic index videos and ultrasound index videos. The swallowing test food, the swallowing test result, and a symbol indicating the type of video are incorporated into an index number indicating the order in which the swallowing timing was detected. If it is assumed that the endoscopic index videos 44a to 44e are extracted in the order in which the swallowing timing was detected, then 44a will be a normal swallow of saliva and will be "001saS1", 44b will be a normal swallow of milk and will be "002miS1", 44c will be an unknown swallow of colored water and will be "003cwU1", 44d will be aspiration of pudding and will be "004puA1", and 44e will be an unknown but normal swallow and will be "005unS1". If the ultrasound index videos 54a to 54e synchronized with each other also have similar results, then 54a will be "001saS2", 54b will be "002miS2", 54c will be "003cwU2", 54d will be "004puA2", and 54e will be "005unS2".

[0060] The display control unit 22 controls switching of the screen displayed on the display 14. When a swallowing test is being performed, a real-time image observed by the user is displayed as an observation screen, and when the captured video or the like is to be displayed after the imaging is completed, the display control unit 22 displays the video as a playback screen. Playback on the observation screen is performed automatically or by user operation.

[0061] In the video editing unit 30, the process from acquiring the endoscopic video 41 and the ultrasound video 51 to displaying the endoscopic index video 44 and the ultrasound index video 54 on the screen is automatically performed. When the imaging by the endoscope 11a and the ultrasound examination terminal 12a is completed, the display on the display 14 is switched from the observation screen to a playback screen. On the playback screen, the endoscopic index video 44 extracted from the acquired endoscopic video 41 and the ultrasound index video 54 extracted from the ultrasound video 51 are displayed.

[0062] When a tag or information is added to one of the synchronized endoscopic index video 44 and ultrasound index video 54, the tag or information is added to the other synchronized video. For example, when a tag with "swallowing timing" is added to any frame image of the endoscopic index video 44, the corresponding frame image of the synchronized ultrasound video 51 is also tagged with "swallowing timing" by the synchronization process.

[0063] The synchronization contents are synchronized with each corresponding frame image in units of 1 / 100th of a second based on the shooting time. In addition to the tagging contents, the classification results of the swallowing classification unit 35 and the editing results of the video editing unit 30 are also reflected. In addition, any content designated by the endoscope video signal and the ultrasound video signal may not be synchronized.

[0064] As shown in Fig. 11, the endoscopic index video 44 and the ultrasound index video 54 on the playback screen displayed on the display 14 are simultaneously displayed and synchronously played back as two types of videos, as in Fig. 5. This allows simultaneous observation of the pharynx and esophagus, and allows continuous and simultaneous observation of the movement of food F from the pharynx to the esophagus. The playback screen is provided with a play button 61, a fast-rewind button 62, a fast-forward button 63, a pause button 64, a seek bar 65, a slider 66, a repeat playback button 67, and a screen switching button 68 for the user to operate the video playback, and the user can use these functions via the user interface 15.

[0065] As shown in Fig. 12, the endoscopic index video 44 and the ultrasound index video 54 on the playback screen displayed on the display 14 may be switched from simultaneous playback in Fig. 11 by using a screen switching button 68 to be displayed separately as in Fig. 6. In the case of separate display, a video information display field 69 may be expanded to further display the swallowing type and index number assigned by the swallowing classification unit 35. The index number may be used as the video title of the endoscopic index video 44.

[0066] The user can confirm, add, or modify information on the endoscopic index video 44 and the ultrasound index video 54 displayed in the video information display field 69 from the user interface 15 via the input receiving unit 21. For example, the information can include findings obtained by playing the video, and characteristics such as the patient's gender and age group. In addition, the automatically classified type of swallowing test food and the presence or absence of aspiration can be edited.

[0067] The user can edit the extraction range of the index video in addition to editing the information in the video information display field 69. If, as a result of reviewing on the playback screen, the swallowing timing is erroneously detected or insufficiently extracted, the endoscopic index video 44 and the ultrasound index video 54 having the swallowing timing may be re-acquired by re-extracting or re-examining the endoscopic video 41 stored in the temporary storage memory 31, including manual extraction.

[0068] After checking and editing the various moving image information, the endoscopic index moving image 44 and the ultrasound index moving image 54 are stored in the storage memory 23. At this time, it is preferable to delete the endoscopic moving image 41 and the ultrasound moving image 51 from which they were extracted and stored in the temporary storage memory 31.

[0069] In deep learning for swallowing detection processing of endoscopic video 41 and ultrasound video 51, the tool is trained in advance on the features of images to be detected, and the tool calculates the probability that an analyzed frame image is a detection target, and a frame image with a probability equal to or greater than a threshold is determined to be a detection target. The detection target is the pharynx that captures food F, and the movement of food F is tracked to detect and tag endoscopic swallowing frame image 42 or ultrasound swallowing frame image 52. Note that deep learning requires learning according to the type of food F used in the swallowing test.

[0070] The pharyngeal phase swallowing timing recorded by the endoscopic video 41 is the dynamics of the pharynx when the patient swallows food F, but the detection does not need to be based on deep learning that recognizes food F, and swallowing detection processing may be performed based on the characteristics of the image during swallowing or changes between previous and next frame images. Specifically, swallowing detection processing may be performed using a swallowing detection algorithm that uses "pixel value difference between previous and next frame images," "amount of blurring of frame images," "number of key points in an image," etc. Regardless of the swallowing detection processing, frame images determined to be during swallowing are tagged with "swallowing timing detected."

[0071] In a detection algorithm based on "pixel value difference between previous and next frame images," since the subject moves a lot between frame images during swallowing, the simple pixel value difference between previous and next frame images is calculated. The swallowing timing detection unit 33 calculates the pixel value difference between two chronologically adjacent frame images. As a function for calculating the difference (simple difference value), it is preferable to use AbsDiff, which is included in OpenCV (registered trademark) used for image analysis.

[0072] In the detection algorithm based on the "amount of blur in frame images," since the image during swallowing has a large blur of the subject between frame images, an edge amount that indicates the amount of blur between frame images is calculated. The swallowing timing detection unit 33 calculates the edge amount related to two frame images that are successive in time series. As a function for calculating the edge amount, it is preferable to use the Variance Of Laplacian installed in OpenCV (registered trademark) used for image analysis.

[0073] In the detection algorithm based on the "number of key points", the image during swallowing has a large subject blur between frame images, so the edges of the frame images become unclear and the number of key points decreases. A key point refers to a feature point that is a corner with a large edge amount among edges extracted by expressing a frame image with a line, and the swallowing timing detection unit 33 calculates the number of key points related to the frame image. It is preferable to use the Count Key Point function installed in OpenCV (registered trademark) used for image analysis as a function for calculating the number of key points. Note that, for feature point extraction, AKAZE (Accelarated KAZE) installed in OpenCV (registered trademark) used for image analysis is used. In the feature point extraction in this embodiment, it is preferable to recognize the part with a high edge amount in the image (part recognized as a "corner" or "angle").

[0074] The real-time display mode will be described. Real-time images not undergoing synchronization processing during imaging by the endoscope system 11 and the ultrasound system 12 are displayed side by side on the display 14. A video signal captured by starting an endoscopic examination of the pharynx by the endoscope system 11 is sent to the display control unit 22 via the endoscope video processing unit 24 of the processor device 13. A video signal captured by starting an endoscopic examination of the esophagus by the ultrasound system 12 is sent to the display control unit 22 via the ultrasound video processing unit 25 of the processor device 13. The display control unit 22 displays the video signal as it is on the display 14 as a real-time image. When recording a video of the swallowing examination captured, it is preferable that the user decides the start and end using a switch or the like provided on the endoscope 11a and the ultrasound examination terminal 12a.

[0075] FIG. 13 is a flow chart showing a series of steps for acquiring index videos of a swallowing test using two systems, endoscopic video capture and ultrasonic video capture. The following describes a flow for extracting and playing an endoscopic index video 44 and an ultrasonic index video 54 in a swallowing test. The user acquires an endoscopic video 41 that captures the state of a swallowing test in the pharynx captured by the endoscope 11a during a real-time display mode, and an ultrasonic video 51 that captures the state of a swallowing test in the esophagus captured by the ultrasonic examination terminal 12a. After the swallowing test is completed, the mode is switched to a normal video synchronization mode or an index video synchronization mode. In either case, the acquired endoscopic video 41 is transmitted to the video editing unit 30 via the endoscopic video processing unit 24, and the ultrasonic video 51 is transmitted to the video editing unit 30 via the ultrasonic video processing unit 25. The endoscopic video 41 and the ultrasonic video 51 transmitted to the video editing unit 30 are stored in the temporary storage memory 31, and then synchronized by the video synchronization unit 32.

[0076] In the normal video synchronization mode, the endoscopic video 41 and the ultrasound video 51 after the synchronization process are displayed on the display 14, and the swallowing timing is observed from two types of images by performing synchronous playback. In the index video synchronization mode, the synchronized endoscopic video 41 and the ultrasound video 51 are analyzed by the swallowing timing detection unit 33 to identify the pharyngeal stage swallowing frame image group 45 or the esophageal stage swallowing frame image group 55, which are frame images capturing the swallowing timing. From the endoscopic video 41 and the ultrasound video 51 transmitted to the index video extraction unit 34 after the swallowing timing detection unit 33, the esophageal stage swallowing frame image group 55 and frame images for a certain period of time before and after the esophageal stage swallowing frame image group 55 are extracted as the endoscopic index video 44 and the ultrasound index video 54 from the pharyngeal stage swallowing frame image group 45. The extracted endoscopic index video 44 and the ultrasound index video 54 are transmitted to the swallowing classification unit 35, where the swallowing examination type is classified and an index number is assigned by machine learning. Thereafter, the endoscopic index video 44 and the ultrasound index video 54 are transmitted to the display control unit 22 and displayed on the display 14. By performing synchronous playback, the swallowing timing can be observed in a short period of time from the two types of images.

[0077] After performing observation by synchronous playback in the normal video synchronization mode and index video synchronization mode, it is possible to confirm the type of swallowing examination and perform editing such as additions and corrections. Each video after editing of the video information is stored in the storage memory 23 in the processor device 13. In this case, it is preferable to delete the endoscopic video 41 and the ultrasound video 51 stored in the temporary storage memory 31 if they are not needed. If the number of acquired videos is insufficient due to extraction failure or the like, or if the image quality is poor, the examination is repeated and videos are acquired again.

[0078] In addition to the above-mentioned esophageal swallowing observation, the ultrasound examination may also perform aspiration and residue evaluation of the pharynx and a movement evaluation of the swallowing-related muscles. The target area is observed with the ultrasound examination terminal 12a, and an ultrasound image signal is acquired. It is preferable to analyze the ultrasound image signal to detect the swallowing timing and extract an index video. In the swallowing observation of the ultrasound examination, the swallowing detection process is mainly an esophageal swallowing detection process, but since the observation area can be easily changed in the non-invasive ultrasound examination, it is also possible to perform oral swallowing detection process and pharyngeal swallowing detection process of food F located in a position as shown in FIG. 2. Note that since the depth from the body surface and the shape of the body surface differ for each observation area, multiple ultrasound examination terminals 12a may be used depending on the observation content of the ultrasound examination.

[0079] In the evaluation of aspiration and residue in the laryngopharynx, the same observation site as that of the swallowing endoscope is used, and the aspiration and residue of food and drink in the pharyngeal stage is evaluated. The ultrasound examination terminal 12a acquires an ultrasound video 51 of the laryngopharynx, and the swallowing timing is detected by learning the image during swallowing using deep learning. The detected swallowing timing can be synchronized with the endoscopic video 41 to extract an ultrasound index video 54. The type and results of swallowed food from the endoscopic examination may be linked to the aspiration and residue evaluation from the ultrasound examination and displayed on the screen.

[0080] In the dynamic evaluation of swallowing-related muscles, the morphology of the tongue, hyoid bone area, or laryngeal muscles is observed to evaluate the swallowing movement. Observation of the tongue and hyoid bone area is an examination of swallowing timing in the oral phase, and observation of the laryngeal muscles is an examination of swallowing timing in the pharyngeal phase. The ultrasound examination terminal 12a acquires an ultrasound video 51, and the swallowing timing is detected by learning images of swallowing in the oral or pharyngeal phase using deep learning. The ultrasound index video 54 can be extracted by synchronizing the detected swallowing timing with the endoscopic video 41. The type and results of swallowed food obtained by endoscopic examination and the dynamic evaluation of swallowing-related muscles obtained by ultrasound examination may be linked and displayed on the screen. [Second embodiment]

[0081] In this embodiment, instead of the ultrasound system 12 of the first embodiment, a radiation observation system 16 having a radiation observation device 16a is used to obtain video signals and observe a swallowing test of a patient, as shown in Fig. 14. The endoscopic video 41 and the radioscopic video 71 of this embodiment will be described below. Note that the contents that overlap with the above embodiment will be omitted.

[0082] The radiation observation system 16 is composed of a radiation observation device 16a. The radiation observation device 16a captures an image of a patient swallowing food F containing a contrast agent, and obtains a radioscopic video 71. The processor device 13 also realizes a function of a radioscopic video processing unit (not shown) that receives the radioscopic video 71 captured by the radiation observation system 16 and sends it to the video editing unit 30.

[0083] As shown in Fig. 15, the endoscopic video 41 and the radioscopic video 71, which have been synchronized by the video synchronization unit 32, can be transmitted to the display control unit 22, thereby enabling simultaneous display and synchronous playback on the display 14. The radioscopic video 71 can display a continuous series of swallowing actions, including the oral phase in which food F is sent from the oral cavity to the pharynx by the tongue, the pharyngeal phase in which food F is sent from the pharynx to the esophagus, and the esophageal phase in which food F is sent from the esophagus to the stomach, as shown in Fig. 2. For this reason, it is preferable to photograph the patient from the side and display on the display 14 an image including the oral cavity, pharynx, and esophagus.

[0084] As shown in Figure 16, the series of swallowing frame images 73 included in the radiological video 71, which captures the oral phase to the esophageal phase, includes the entire pharyngeal phase swallowing frame image group 45 that captures swallowing in the pharyngeal phase, and therefore the endoscopic index video 44 and the radiological index video 74 extract the series of swallowing frame images 73 and frame images of a certain period before and after it from the endoscopic video 41 and the radiological video 71.

[0085] In the swallowing classification unit 35, in a swallowing examination using the radiation observation system 16, a contrast agent is used and therefore the type of food F is different from that in the first embodiment. Therefore, the classification of the swallowing type is different from that in the first embodiment. The index number is suffixed with (1) for the endoscopic index video 44 and (3) for the radiological index video 74 according to the type of video. After classification, the endoscopic index video 44 and the radiological index video 74 are transmitted to the display control unit 22 and displayed on the display 14, allowing synchronous playback.

[0086] [Third embodiment] In the above embodiment, different imaging systems were combined with the endoscope system 11 to record the swallowing test using two systems, but in this embodiment, a swallowing test is observed by combining three types of imaging systems.

[0087] 17, a medical image processing system 10 performs observation of a swallowing test by integrating three systems using an endoscope system 11, an ultrasound system 12, and a radiation observation system 16. A review of the swallowing test of this embodiment will be described below. Note that the contents that overlap with the above embodiment will be omitted.

[0088] 18, an endoscopic video 41, an ultrasound video 51, and a radioscopic video 71, which have been synchronized by the video synchronization unit 32, can be transmitted to the display control unit 22, thereby enabling simultaneous display and synchronous playback on the display 14. A screen switching button 68 may be used to switch from simultaneous display of the three types to individual display as shown in FIG.

[0089] As shown in FIG. 19, the series of swallowing frame images 73 capturing the period from the oral phase to the esophageal phase included in the radiological video 71 includes the entire pharyngeal phase swallowing frame image group 45 capturing swallowing during the pharyngeal phase and the entire esophageal phase swallowing frame image group 55 capturing swallowing during the esophageal phase, and therefore the endoscopic index video 44, ultrasound index video 54, and radiological index video 74 extract the series of swallowing frame images 73 and frame images of a certain period before and after it from the endoscopic video 41, ultrasound video 51, and radiological video 71.

[0090] In each embodiment, the hardware structure of the processing units that execute various processes, such as the display control unit 22, the storage memory 23, the endoscopic video processing unit 24, the ultrasonic video processing unit 25, the video editing unit 30, the temporary storage memory 31, the video synchronization unit 32, the swallowing timing detection unit 33, the index video extraction unit 34, and the swallowing classification unit 35, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0091] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0092] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (Circuitry) in the form of a combination of circuit elements such as semiconductor elements, etc. Also, the hardware structure of the memory unit is a storage device such as an HDD (Hard Disc Drive) or an SSD (Solid State Drive). [Explanation of symbols]

[0093] 10 Medical image processing system 11 Endoscope system 11a Endoscope 11b Endoscope tip 12 Ultrasound Systems 12a Ultrasound inspection terminal 12b Ultrasonic inspection terminal contact part 13 Processor unit 14 Display 15 User Interface 16 Radiation Monitoring System 16a Radiation Observation Device 20 Central Control Unit 21 Input receiver 22 Display control unit 23 Storage memory 24 Endoscope video processing unit 25 Ultrasonic video processing unit 30 Video Editing Department 31 Temporary storage memory 32 Video Synchronization Section 33 Swallowing timing detection unit 34 Index video extraction unit 35 Swallowing Classification Department 41 Endoscopy Video 42 Endoscopic swallowing frame image 43 Endoscopic non-swallowing frame image 44 Endoscopy Index Video 44a Endoscopy Index Video 44b Endoscopy Index Video 44c Endoscopy Index Video 44d Endoscopy Index Video 44e Endoscopy Index Video 45 Pharyngeal swallowing frame images 51 Ultrasound Video 52 Ultrasound swallowing frame image 53 Ultrasound non-swallowing frame image 54 Ultrasound Index Video 54a Ultrasound Index Video 54b Ultrasound Index Video 54c Ultrasound Index Video 54d Ultrasound Index Video 54e Ultrasound Index Video 55 Esophageal swallowing frame images 60 Video information display field 61 Play button 62 Rewind button 63 Fast forward button 64 Pause button 65 Slider 66 Seek Bar 67 Repeat Play Button 68 Screen switching button 71 Radiographic Video 73 Swallowing Frame Image Series 74 Radiation Index Video F Food Th Throat Nc nasal cavity R position

Claims

1. A processor is provided. The processor, receiving an endoscopic video signal obtained by imaging a swallowing test of the patient's throat using an endoscope inserted through the patient's nasal cavity; receiving an ultrasound image signal of the esophagus, airway, larynx, tongue, or hyoid bone surrounding area obtained by an ultrasound examination terminal placed on the patient's neck; A medical image processing system that simultaneously displays endoscopic video signals and ultrasound video signals on a display.

2. The processor, When the endoscope image signal and the ultrasound image signal are acquired, an imaging time based on a time of the processor is recorded; performing a synchronization process for synchronizing the endoscopic video signal and the ultrasonic video signal based on the imaging time; 2. The medical image processing system according to claim 1, wherein the synchronously processed endoscopic video signal and ultrasonic video signal are reproduced on the display in synchronization with each other.

3. 3. The medical image processing system according to claim 2, wherein the endoscopic video signal and the ultrasonic video signal are reproduced automatically or by a user operation.

4. The processor, When the endoscope image signal and the ultrasound image signal are acquired, an imaging time based on a time of the processor is recorded; performing synchronization processing of the endoscope video signal and the ultrasound video signal based on the imaging time; Detecting a swallowing timing from the endoscope video signal, tagging the swallowing timing, and extracting an endoscope index video; The ultrasound image signal having the same imaging time as the endoscopic image signal at the swallowing timing is tagged, and an ultrasound index video is extracted. The medical image processing system according to claim 1 , wherein the endoscopic index video and the ultrasound index video are reproduced on the display.

5. The processor, When the endoscope image signal and the ultrasound image signal are acquired, an imaging time based on a time of the processor is recorded; performing synchronization processing of the endoscope video signal and the ultrasound video signal based on the imaging time; Detecting a swallowing timing from the endoscope video signal; Detecting non-swallowing timing from the ultrasound image signal; Among the endoscopic video signals at the swallowing timing, tagging is performed on the endoscopic video signals having the same imaging time as the ultrasound video signals other than the non-swallowing timing, to extract an endoscopic index moving image; Among the ultrasound image signals other than the non-swallowing timing, the ultrasound image signals having the same imaging time as the endoscopic image signal at the swallowing timing are tagged to extract an ultrasound index moving image; The medical image processing system according to claim 1 , wherein the endoscopic index video and the ultrasound index video are reproduced on the display.

6. 6. The medical image processing system according to claim 4, wherein the endoscopic index moving image and the ultrasonic index moving image are reproduced automatically or by a user operation.

7. The processor, Detecting a swallowing timing as normal swallowing, erroneous swallowing, or unclear swallowing from the endoscopic video signal and the ultrasonic video signal; 7. The medical image processing system according to claim 4, further comprising a step of tagging the frame image in which the swallowing timing is detected with a swallowing timing detection tag.

8. the endoscope video signal and the ultrasound video signal are composed of a group of successive frame images, The processor, The swallowing timing of the endoscope video signal is determined by calculating the amount of blur between frame images, calculating key points based on the frame images, or calculating a difference in pixel values ​​between the frame images. The medical image processing system according to claim 4 , wherein machine learning is used to determine whether or not the swallowing timing is present in the ultrasound image signal.

9. The processor, 9. The medical image processing system according to claim 4, further comprising: analyzing at least one of the endoscopic index video and the ultrasound index video to identify the type of the swallowing test.

10. The processor, The type of swallowing test is classified according to the type of swallowing test food and the swallowing result, The medical image processing system according to claim 9 , further comprising: assigning index numbers to the endoscopic index video and the ultrasound index video based on the classification.

11. The processor, In acquiring the endoscopic video signal and the ultrasonic video signal, imaging is performed at a frame rate of 60 fps; 11. The medical image processing system according to claim 1, wherein the imaging time is counted in units of 1 / 100 seconds.

12. The processor, 2. The medical image processing system according to claim 1, wherein the endoscopic video signal and the ultrasonic video signal are simultaneously displayed on a display in real time without synchronization.

13. The processor, receiving a radiographic image signal obtained by observing a swallowing examination of the patient using a radiation observation device instead of the observation using the ultrasound examination terminal; 2. The medical image processing system according to claim 1, wherein the endoscopic image signal and the radioscopic image signal are displayed on the display.

14. The processor, receiving a radiographic image signal obtained by observing a swallowing examination of the patient using a radiation observation device in addition to the observation using the ultrasound examination terminal; 2. The medical image processing system according to claim 1, wherein the endoscopic image signal, the ultrasonic image signal, and the radioscopic image signal are displayed on the display.

Citation Information

Patent Citations

  • Production of double layered steel ingot

    JP1983023535A

  • Ultrasonic evaluation system for physical property in swallowing

    JP2001218769A

  • Ultrasonograph

    JP2004121488A

  • Test food for assessment of ingestion function and test food kit for assessment of ingestion function

    JP2012193144A

  • Medical image diagnostic device and medical image processor

    JP2015211831A