Medical image processing system and method of operation thereof

Abstract

To provide a medical image processing system capable of efficiently observing deglutition by automatically extracting index moving images out of moving images obtained by capturing deglutition check and minimizing the time for observation of deglutition with the moving images, and an operation method of the system.SOLUTION: A medical image processing system analyzes a moving image file 41 acquired by deglutition check and determines the presence or absence of deglutition timing being deglutition in a pharyngeal phase. When deglutition timing is detected, the system extracts, screen-displays, and automatically plays back index moving images with the deglutition timing as the center.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a medical imaging system and a method of operation thereof. [Background technology]

[0002] When an endoscope is inserted into a subject for observation, video recording may be performed to record the movement inside the subject or to prevent missing still images. The endoscope operator presses the record button on the video recorder to start recording before starting observation or inserting the tip of the endoscope into the subject, and continues recording until the end of observation or the removal of the tip of the endoscope from the subject, so the recording time of the acquired video file is long. In a swallowing endoscope, a series of swallowing patterns is recorded in video, but the part used for diagnosis is almost always less than 5 seconds for one swallow, and multiple swallowing reactions are observed during one examination. Since the recording time of a video recording one swallow is several minutes or more, there is a loss when checking the target swallow later. Therefore, when diagnosing swallowing, it is necessary to selectively play only the swallowing part of the acquired video file.

[0003] However, in swallowing diagnosis, it is inefficient to search for the target part by specifying the playback time or fast-forwarding the video, and the management of video files tends to be cumbersome. Also, since the shooting part is roughly the same, it is difficult to know which video is the result of what was done. Also, if the playback time is long, it takes time to look back.

[0004] Patent Document 1 describes a method of creating chapter images by performing a freeze operation at any timing during endoscopic image acquisition, and then playing back the chapter image from the acquisition point after observation is completed, or comparing it with images captured in the past. Patent Document 2 describes a method of evaluating the characteristics of a subject's swallowing process when ingesting food by linking the characteristics of the swallowing process to a vibration sensor and an imaging technique. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2018 / 043585 [Patent Document 2] Special Publication No. 2017-510368 Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, chapter images are acquired by user operation, so there is a possibility that some images may be missed or the playback position may be shifted. In addition, there is no description of observing swallowing or the pharynx. In Patent Document 2, swallowing dynamics are sensed and swallowing characteristics are evaluated using an image technique, but the method involves sensing the vibrations of swallowing and then photographing, and there is no description of observing swallowing using images alone. In light of the above, it is desirable to minimize the time required to observe swallowing using video and to efficiently observe swallowing by accurately detecting swallowing and extracting images of the swallowing without a time lag.

[0007] An object of the present invention is to provide a medical image processing system and an operating method thereof that can automatically extract an index moving image from a moving image captured during a swallowing test and automatically play the index moving image after the swallowing test is completed. [Means for solving the problem]

[0008] The medical image processing system of the present invention includes a processor that receives a video signal recorded by an endoscope during a swallowing examination, analyzes the video signal to determine whether or not there is a swallowing timing, treats a frame image of the swallowing timing as a swallowing frame image tagged with swallowing timing detection, and extracts an index video including the swallowing frame image from the video signal.

[0009] The index moving image is preferably composed of a swallowing frame image group including a swallowing frame image and a frame image group for a certain period of time subsequent to the swallowing frame image group.

[0010] The group of frame images for a certain period of time is preferably non-swallowing frame images that are before the start and after the end of the group of swallowing frame images and are not tagged with swallowing timing detection.

[0011] The video signal includes frame images to be analyzed, and it is preferable that the processor determines whether or not there is a swallowing timing by either calculating the amount of blur in the frame images, calculating key points based on the frame images, or using the difference in pixel values ​​between the frame images.

[0012] The processor preferably analyses the index video to identify the type of swallowing test.

[0013] The processor preferably assigns an index number to the index video for use in video searching.

[0014] It is preferable that the processor automatically plays the index video when it is displayed on the screen without any user operation.

[0015] It is preferable that the processor displays a list of the multiple index videos on the display and automatically plays them all at once or continuously.

[0016] It is preferable that the processor displays at least one of the type of swallowing test or the index number when displaying the index moving image on the screen.

[0017] The processor preferably joins together a plurality of index videos to create a composite index video that enables the index videos to be played continuously.

[0018] It is preferable that the processor determines whether or not there is a swallowing timing by performing voice recognition during swallowing.

[0019] In a method of operating the medical image processing system of the present invention, the processor includes a step of receiving a video signal recording a swallowing examination by an endoscope, a step of analyzing the video signal to determine the presence or absence of a swallowing timing and converting a frame image of the swallowing timing into a swallowing frame image tagged with swallowing timing detection, and a step of extracting an index video including the swallowing frame image from the video signal. Effect of the Invention

[0020] Index videos can be automatically extracted from videos taken during swallowing tests and automatically played back after the swallowing test is completed, allowing for efficient observation of swallowing. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is a schematic diagram illustrating a medical image processing system. [Diagram 2] FIG. 1 illustrates the stages of swallowing food. [Diagram 3] FIG. 2 is a block diagram showing the functions of the medical image processing apparatus. [Figure 4] 11 is an explanatory diagram for detecting swallowing timing from a video signal and extracting an index moving image. FIG. [Diagram 5] 1A to 1C are image diagrams showing observed images of the pharynx when food is swallowed and when it is not swallowed. [Figure 6] 1A and 1B are image diagrams showing the extraction of multiple index moving images from a video signal. [Figure 7] FIG. 13 is an image diagram showing the swallowing classification results and allocation to index numbers. [Figure 8] FIG. 13 is a diagram showing an image when index videos are displayed in a list on a screen. [Figure 9] FIG. 13 is an image diagram showing one index video displayed on a screen. [Figure 10] FIG. 13 is a diagram showing an image of a search screen for an index video. [Figure 11] FIG. 13 is an image diagram showing index videos being combined together. [Figure 12]1A and 1B are explanatory diagrams and image diagrams showing an imaging method in a swallowing test. [Figure 13] FIG. 13 is an explanatory diagram showing a method of determining swallowing based on the pixel value difference between previous and subsequent frame images. [Figure 14] FIG. 13 is an explanatory diagram showing a method for determining swallowing based on blurring of frame images. [Figure 15] FIG. 13 is an explanatory diagram showing a method of determining swallowing based on the number of key points in a frame image. [Figure 16] 11 is a flowchart for extracting an index moving image from a video signal. [Figure 17] FIG. 11 is a block diagram showing functions of a swallowing timing detection unit realized in the second embodiment. [Figure 18] FIG. 13 is a diagram showing an image in which an extracted index moving image and a past index moving image are simultaneously displayed, as realized in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] [First embodiment] As shown in FIG. 1, the medical image processing system 10 includes a medical image processing device 11, a database 12, an endoscope system 13, a display 14, and a user interface 15. The medical image processing device 11 is electrically connected to the database 12, the endoscope system 13, the display 14, and the user interface 15. The database 12 is a device that stores acquired images and can transmit and receive data to and from the medical image processing device 11, and may be a recording medium such as a USB (Universal Serial Bus) or a HDD (Hard Disc Drive). The medical image processing device 11 acquires images captured in a swallowing test from an endoscope 13a constituting the endoscope system 13. The user interface 15 is an input device for inputting settings to the medical image processing device 11, and includes a keyboard, a mouse, and the like.

[0023] The endoscope 13a is a swallowing endoscope that is inserted through the patient's nasal cavity and illuminates the area around the pharynx with illumination light to observe and photograph swallowing. Since swallowing is a dynamic process, video is acquired during the swallowing examination. Unless otherwise specified, white light is used as illumination light for the swallowing photography, and a video signal of 60 frames per second (60 fps) is acquired.

[0024] As shown in FIG. 2, swallowing is a series of movements in which food F put in the mouth is chewed, swallowed, and sent to the esophagus. The progression of swallowing includes an "oral stage" in which food F put in the mouth is sent from the tongue to the pharynx, a "pharyngeal stage" in which food F is sent from the pharynx to the esophagus, and an "esophageal stage" in which food F is sent from the esophagus to the stomach. In this embodiment, the "pharyngeal stage" of swallowing is set as the swallowing timing, and the pharynx is observed. The food F is a plurality of liquids or solids that are not harmful to the human body when swallowed and have different ease of swallowing. For example, milk, a colored aqueous solution, and pudding are used.

[0025] In a swallowing test, the patient takes food F into the mouth, swallows it at the pharynx, and it passes through the esophagus and into the stomach. fart The state of the food being delivered is recorded. Even if the patient cannot swallow food F, the state is recorded. In a swallowing test, it is preferable to check multiple swallows in succession, rather than one swallow at a time. For example, in a swallowing test, the patient swallows food F in the following order: a colored water solution, milk, the patient's own saliva, and pudding.

[0026] 3, in the medical image processing device 11, programs related to processing such as image processing are stored in a program memory (not shown). In the medical image processing device 11, a central control unit 20 constituted by an image control processor operates the programs in the program memory to realize the functions of an index moving image creating unit 30, an image receiving unit 21, a display control unit 22, an input receiving unit 23, and a storage memory 24. In addition, with the realization of the function of the index moving image creating unit 30, the functions of a temporary storage area 31, a swallowing timing detecting unit 32, an index moving image extracting unit 33, and a swallowing classifying unit 34 are realized.

[0027] The image receiving unit 21 receives a moving image file 41 that records a swallowing examination performed by the endoscope 13a, and transmits the moving image file 41 to the index moving image creating unit 30. The index moving image creating unit 30 extracts an index moving image 42 from the moving image file 41, and a display control unit 22 The index video 42 is transmitted to the display 14 via the input terminal 23. The display control unit 22 controls the display of the index video 42 on the display 14. The input receiving unit 23 is connected to the user interface 15. The storage memory 24 can achieve a storage function independently, but can also be connected to the database 12 to transmit and receive data. The video file 41 is received after the swallowing test is completed, but the video signal may be processed in real time during the swallowing test, before the video file 41 is created.

[0028] As shown in Fig. 4, in the index video creation unit 30, scenes in which swallowing does not occur are excluded from a video file 41, and an index video 42 is extracted in order to efficiently observe swallowing. The index video 42 is composed of a swallowing frame image group 43 and a group of frame images for a certain period of time preceding and succeeding the swallowing frame image group 43. The swallowing frame image group 43 is composed of a plurality of consecutive swallowing frame images 44, and the swallowing frame images 44 are frame images that capture the timing of swallowing. The arrows in Fig. 4 represent the progression of time, and when the video file 41 is played, the frame images are played and displayed in the direction of the arrow. A single frame image is one image, and when this is combined into a group of consecutive images, a video is formed.

[0029] The temporary storage area 31 temporarily stores the moving image file 41 sent to the index moving image creation unit 30. The temporarily stored moving image file 41 is transmitted to the swallowing timing detection unit 32.

[0030] The swallowing timing detection unit 32 performs a swallowing detection process in which the video file 41 is image-analyzed using a tool dedicated to machine learning such as deep learning, and determines whether or not there is a swallowing timing, which is the dynamic state when food F passes through the pharynx. The portion corresponding to the swallowing timing is a swallowing frame image group 43, which is a series of frame images in which food F is included in the video file 41. The video file 41 that has undergone the swallowing detection process is transmitted to the index video extraction unit 33. It is preferable that the video file 41 in which the swallowing timing is not detected is stored in the storage memory 24, and deleted from the temporary storage area 31.

[0031] 5 shows a swallowing frame image 44 which constitutes a video file 41 and is a frame image showing food F passing through the pharynx, and a non-swallowing frame image 45 which constitutes a video file 41 and is a frame image not showing food F passing through the pharynx. The swallowing frame image 44 is a frame image in which food F is recognized in the swallowing detection process and is tagged, for example, with "swallowing timing detection." The non-swallowing frame image 45 is a frame image not tagged with "swallowing timing detection" in which the swallowing timing is not recognized by machine learning in the swallowing detection process, i.e., food F is not captured by deep learning.

[0032] The index video extraction unit 33 extracts an index video including a frame tagged with "swallowing timing detection" from the video file 41. Specifically, the index video extraction unit 33 extracts the swallowing frame image group 43 contained in the video file 41 and a group of frame images for a certain period of time consecutive to the start and end of the swallowing frame image group 43 as the index video 42. The swallowing frame image group 43 is extracted from the video file 41 using frame images tagged with "swallowing timing detection". The certain period of time is a time of about 3 or 5 seconds set in advance by the user, and it is preferable that the movement of the pharynx before and after the passage of food F is captured.

[0033] 6, when swallowing is not always observed only once in one video file 41 and multiple swallowing timings are detected, multiple index videos 42 are extracted from one video file. For example, when different swallowing timings 43a and 43b are detected from one video file 41 by the swallowing timing detection unit 32, the index video extraction unit 33 individually extracts an index video 42a having the swallowing timing 43a and an index video 42b having the swallowing timing 43b.

[0034] The swallowing classification unit 34 analyzes the index video 42 and assigns the type of swallowed food, the result, and the index number of the swallowing test in the index video 42. The classification result is a classification of the type of swallowed food F, such as swallowing saliva, swallowing milk, swallowing a colored aqueous solution, eating pudding, etc., and also whether the swallowing was normal or involved aspiration. A score evaluation that assigns points according to the degree of aspiration may be used. These are automatically classified based on data learned by machine learning. It is preferable that the automatically classified information on the type of swallowing test food is assigned to the index video 42. The index number is a number used when a user searches the index video 42 stored in the storage memory 24, etc., and is preferably a string of characters with a small number of digits that does not overlap, such as an alphanumeric representation of the order in which the swallowing timing was detected and the type of swallowing test food. In this case, the type of swallowing is classified as saliva (sa), milk (mi), colored water (cw), pudding (pu), and unknown (un), and in addition, normal swallowing (S) or aspiration (A) is classified. The classified index moving image 42 is transmitted to the display control unit 22 .

[0035] 7, for example, consecutively extracted index videos 42a, 42b, 42c, 42d, and 42e are classified into swallowing types by the swallowing classification unit 34, and the classification results are incorporated into an index number indicating the order in which the swallowing timings were detected. If the index videos 42a to 42e are extracted in the order in which the swallowing timings were detected, then 42a is a normal swallow of saliva and is labeled "001saS," 42b is a normal swallow of milk and is labeled "002miS," 42c is a normal swallow of colored water and is labeled "003cwS," 42d is aspiration of pudding and is labeled "004puA," and 42e is unknown and is a normal swallow "005unS."

[0036] The display control unit 22 controls switching of the screen displayed on the display 14. When a swallowing test is performed and the pharynx is being photographed with the endoscope 13a, a real-time image observed by the user is displayed as an observation screen, and when the photographing is completed and the acquired video or the like is to be displayed, the display control unit 22 displays the real-time image observed by the user as an observation screen.

[0037] The index moving image creating unit 30 automatically performs processes from obtaining the moving image file 41 to transmitting the index moving image 42 to the display control unit 22. When the imaging by the endoscope 13a is completed, the display on the display 14 switches from the observation screen to a playback screen. On the playback screen, the index moving image 42 extracted from the obtained moving image file 41 is displayed.

[0038] As shown in FIG. 8, the acquired index videos 42 are displayed in a list on the playback screen of the display 14. The index videos 42 displayed in a list are automatically played back, and the user can review the swallowing timing and compare the index videos 42 without any operation. This automatic playback is preferably performed one by one in the order in which they were shot. If the index videos are arranged in the order of 42a, 42b, 42c, ..., they are played in that order. It is preferable that the playback screen simultaneously displays a video information display field 50 that displays various information about the index video 42 being played. The playback screen is provided with a play button 51, a fast-rewind button 52, a fast-forward button 53, a pause button 54, a seek bar 55, a slider 56, and a repeat playback button 57 for the user to operate the video playback.

[0039] In continuous playback of the index videos 42, it is preferable to highlight the video being played. For example, as shown in index video 42b, the frame of the video may be made thicker to make it easier to see. The index video 42 is automatically played during automatic display, and the content displayed in the video information display field 50 can be confirmed and edited by the user. The video information display field 50 displays information such as the type of swallowing and index number assigned by the swallowing classification unit 34, the patient's name and age, the name of the photographer, the video title, and findings. The index number may be used as the video title of the index video 42.

[0040] On the playback screen, pressing the play button 51 allows the index video 42 that has finished automatic playback to be played repeatedly, the fast rewind button 52 allows the user to go back to a scene that was missed, the fast forward button 53 allows the user to increase the video playback speed, and the pause button 54 allows the user to stop playback at any time. The playback status of the video is represented by the position of a slider 56 on a seek bar 55, and the user can freely change the playback point by moving the position of the slider 56 by a user operation such as dragging. When the repeat playback button 57 is selected, the index video 42 that has finished playing will be played repeatedly. When playing continuously, it is preferable that the seek bar 55 display a division of the playback time for each index video 42.

[0041] 9, the acquired index videos 42 can be individually displayed on the display 14. The display can be switched to individual display by the user arbitrarily selecting a target for individual display from the multiple index videos 42 displayed in a list. Automatic playback of the index videos 42 is still performed when switching to individual display.

[0042] The user can confirm, add, or modify information on the index video 42 displayed in the video information display field 50. For example, the user can add features such as findings obtained by playing the index video 42, the gender and age group of the patient, or edit the automatically classified swallowing type, through the user interface 15 via the input receiving unit 23.

[0043] When editing the information in the video information display field 50, the user can also edit the extraction range of the index video 42. If, as a result of reviewing on the playback screen, the swallowing timing is erroneously detected or insufficiently extracted in the index video 42, the index video 42 may be reacquired by re-extracting or re-examining the video file 41 stored in the temporary storage area 31, including manual extraction.

[0044] After the various pieces of moving image information have been confirmed and edited, the index moving image 42 is stored in the storage memory 24 or the database 12. It is preferable to delete the original moving image file 41 stored in the temporary storage area 31.

[0045] 10, when there are many index videos 42 to be played on the playback screen, the desired index video 42 may be searched for by selecting or inputting a keyword. In this case, the display on the display 14 is switched to a search screen for searching. In addition to the index number, the search can be performed using keywords related to the type of dysphagia food, the presence or absence of aspiration, patient information, and findings.

[0046] As shown in FIG. 11, after each index video 42 is saved, a single video may be created by connecting multiple arbitrary index videos 42, so that a video that allows continuous swallowing observation can be played. For example, the index videos 42a to 42e classified in FIG. 7 may be combined to create a composite index video 46, and swallowing may be confirmed continuously in the order in which they were shot. In creating the composite index video 46, index videos 42 of the same type of swallowing, normal swallowing, and aspiration may be combined in addition to the order. Note that an index number is assigned to each swallowing timing, so the index number of the composite index video 46 is a combination of the index numbers of each swallowing timing.

[0047] The swallowing detection process will be described. In deep learning, the tool is trained in advance on the features of the image to be detected, and the tool calculates the probability that the analyzed frame image is the detection target, and a frame image with a probability equal to or higher than a threshold is determined to be the detection target. The detection target is the pharynx that captures food F, and the movement of food F is tracked to detect and tag swallowing frame images 44. Note that deep learning requires learning according to the type of food F used in the swallowing test.

[0048] Swallowing timing is the movement of the pharynx when a patient swallows food F, but its detection does not need to be based on deep learning that recognizes food F; swallowing detection processing can be performed based on the characteristics of the image during swallowing or changes between previous and next frame images. Specifically, swallowing detection processing is performed using a swallowing detection algorithm that uses "pixel value differences between previous and next frame images," "amount of blurring of frame images," "number of key points in an image," etc. Regardless of the swallowing detection processing, frame images that are determined to be in the process of swallowing are tagged with "swallowing timing detected."

[0049] 12, the video is captured with the endoscope tip 13b, which is the tip of the endoscope 13a, located near position R. The frame images constituting the video preferably include anatomical structures such as the epiglottis Eg, glottis Rg, and left and right pyriform sinuses Ps, as shown in example 100. The glottis Rg refers to the space between the left and right folds that constitute the vocal cords.

[0050] In a detection algorithm based on "pixel value difference between previous and next frame images," since the subject moves a lot between frame images during swallowing, the simple pixel value difference between previous and next frame images is calculated. The swallowing timing detection unit 32 calculates the pixel value difference between two chronologically adjacent frame images. It is preferable to use AbsDiff, which is included in OpenCV (registered trademark) used for image analysis, as a function for calculating the difference (simple difference value).

[0051] As shown in FIG. 13, an area (image processing target area 101g) of at least 10 pixels from the center of each frame image in the vertical and horizontal directions is treated as the image processing target area, and a simple difference value is taken, which is the difference in pixel values ​​in the image processing target area 101g between the previous frame image (upper part of FIG. 13, previous frame image 101a, and lower part of FIG. 13, previous frame image 101d) and the subsequent frame image (upper part of FIG. 13, subsequent frame image 101b, and lower part of FIG. 13, subsequent frame image 101e). If this simple difference value is equal to or greater than the first threshold value, the subsequent frame image 101b for which the simple difference value is calculated is determined to be in the process of swallowing. The first swallowing detection algorithm utilizes the fact that the subject (especially the vicinity of the epiglottis Eg) moves vigorously with the swallowing movement, and therefore the simple difference value between frame images is larger than that in the case of not swallowing. The pixel value, which is the value of the first threshold value, is preferably 1 to 255, and can be set arbitrarily.

[0052] A specific example will be described in detail. The upper part of FIG. 13 is an example of calculating a simple difference value between two frame images (a front frame image 101a and a rear frame image 101b) in which no swallowing movement occurs. In a state in which no swallowing movement occurs, the epiglottis Eg is open, and the glottis Rg can be easily observed. In addition, the soft palate (not shown), which is the roof of the oral cavity, and the epiglottis Eg hardly move, and only slightly move due to breathing. Therefore, the movement of the endoscope tip 13b is small, and as shown in the difference example 101c in the upper part of FIG. 13, there is almost no movement or blurring in the entire image. Therefore, the simple difference value between the front frame image 101a and the rear frame image 101b in the image processing target region 101g is less than the first threshold, and the rear frame image 101b is determined to be not swallowing. In the difference example 101c, the difference to be calculated as the simple difference value is illustrated by a line.

[0053] The lower part of FIG. 13 is an example of calculating a simple difference value between two frame images (a front frame image 101d and a rear frame image 101e) before and after a swallowing movement occurs. In a state where a swallowing movement occurs, the soft palate moves and the glottis Rg closes due to the epiglottis Eg, so the endoscope tip 13b moves violently and a large blur occurs in the entire image, and the difference between the frame images becomes large as shown in the difference example 101f in the lower part of FIG. 13. In this case, the simple difference value between the front frame image 101d and the rear frame image 101e in the image processing target region 101g is equal to or greater than the first threshold, and the rear frame image 101e is determined to be in the process of swallowing. As in the upper part, in the difference example 101f, the difference to be calculated as the simple difference value is illustrated by lines. In addition, since the amount of blur between the frame images is large during swallowing, the difference example 101f in the lower part, which is in the process of swallowing, has more difference lines than the difference example 101c in the non-swallowing state.

[0054] In the detection algorithm based on the "amount of blur in frame images," since the image during swallowing has a large blur of the subject between frame images, an edge amount that indicates the amount of blur between frame images is calculated. The swallowing timing detection unit 32 calculates the edge amount related to two frame images that are successive in time series. As a function for calculating the edge amount, it is preferable to use the Variance Of Laplacian installed in OpenCV (registered trademark) used for image analysis.

[0055] As shown in FIG. 14, an area of ​​at least 10 pixels vertically and horizontally from the center of at least one frame image (image processing area 102g) is set as the image processing target, and the Variance Of Laplacian of the frame images (frame image 102a in the upper part of FIG. 14 and frame image 102c in the lower part of FIG. 14) is calculated. workmanThe amount of edge (upper part of FIG. 14, example of edge amount 102b, and lower part of FIG. 14, example of edge amount 102d) is calculated. If this edge amount is equal to or greater than a second threshold, the frame image for which the edge amount was calculated is determined to be during swallowing. The second swallowing detection algorithm utilizes the fact that since the endoscope tip 13b moves vigorously during swallowing, the amount of edge increases as the blur increases. The pixel value, which is the value of the second threshold, is preferably between 1 and 255, and can be set arbitrarily.

[0056] A specific example will be described in detail. The upper part of FIG. 14 is an example of calculating the edge amount of a frame image 102a in which no swallowing movement occurs. In a state in which no swallowing movement occurs, the soft palate and epiglottis Eg hardly move, and the endoscope tip 13b hardly moves either, and as shown in the edge amount example 102b in the upper part of FIG. 14, there is almost no movement or blurring in the entire image. In this case, the edge amount in the image processing target region 102g is less than the second threshold, and the frame image 102a is determined to be not swallowing. In the edge amount example 102b, the part to be calculated as the edge amount is illustrated by a line.

[0057] The lower part of Fig. 14 shows an example of the edge amount calculated for a frame image 102c in which a swallowing movement is occurring. When a swallowing movement is occurring, the soft palate and epiglottis Eg move, causing the endoscope tip 13b to move significantly, resulting in blurring of the entire image. 2 As shown in FIG. 1D, the edge amount becomes large. In this case, the edge amount in the image processing target region 102g is equal to or larger than the second threshold, and the frame image 102c is determined to be during swallowing. As in the upper part, in the edge amount example 102d, the part to be calculated as the edge amount is illustrated by a line. Also, since the amount of blur is large during swallowing, the edge amount example 102d in the lower part, which is during swallowing, has more lines to be used for calculating the edge amount than the example 102b of not swallowing.

[0058] In a detection algorithm based on the "number of key points," images during swallowing have a large subject blur between frame images, which makes the edges of the frame images unclear and reduces the number of key points. A key point refers to a feature point that is a corner with a large edge amount among edges extracted by expressing a frame image with a line, and the swallowing timing detection unit 32 calculates the number of key points related to the frame images. It is preferable to use the Count Key Point function included in OpenCV (registered trademark) used for image analysis as a function for calculating the number of key points.

[0059] As shown in FIG. 15, an area (image processing target area 103g) of at least 10 pixels from the center of at least one frame image in the vertical and horizontal directions is set as the image processing target. As illustrated in FIG. 15, feature points 103c are extracted from the frame images (frame image 103a in the upper part of FIG. 15 and frame image 103b in the lower part of FIG. 15) and the number is calculated. If the number of feature points 103c is equal to or less than a third threshold, the frame image for which the number of feature points is calculated is determined to be in swallowing. The third swallowing detection algorithm utilizes the fact that the tip portion 13b of the endoscope moves violently during swallowing, resulting in large blur. The third threshold value is equal to or greater than 0 and can be set arbitrarily. In addition, if the number of feature points is equal to or less than the third threshold, a swallowing determination value is calculated by multiplying the pixel value by -1, and if the swallowing determination value is less than the threshold, it may be determined that swallowing is in progress. In this case, if the number of feature points exceeds the third threshold, it is determined that not swallowing is in progress.

[0060] For feature point extraction, AKAZE (Accelerated KAZE) installed in OpenCV (registered trademark) used for image analysis is used. In the feature point extraction in this embodiment, it is preferable to recognize parts in the image with a high edge amount (parts recognized as "corners").

[0061] A specific example will be described in detail. The upper part of Fig. 15 is an example of calculating the number of feature points 103c in a frame image 103a in which no swallowing movement occurs. In a state in which no swallowing movement occurs, the endoscope tip 13b hardly moves, and there is almost no movement or blurring in the entire image, so that the number of detected feature points 103c is large, as shown in the frame image 103a in the upper part of Fig. 15. In the example of Fig. 15, when the third threshold is set to 5, the number of feature points 103c in the image processing target region 103g is 30, which exceeds the third threshold, and therefore the frame image 103a is determined to be not swallowing.

[0062] The bottom part of Fig. 15 shows an example of calculating the number of feature points 103c in a frame image 103b in which a swallowing movement is occurring. When a swallowing movement is occurring, the endoscope tip 13b also moves significantly, causing blurring in the entire image, making it difficult to detect the feature points 103c, as shown in the frame image 103b in the bottom part of Fig. 15. In the example of Fig. 15, if the third threshold is set to 5, the number of feature points 103c in the image processing target region 103g is 0, which is less than the third threshold, and therefore the frame image 103b is determined to be in the process of swallowing.

[0063] FIG. 16 is a flowchart showing a series of steps for automatically extracting an index video 42 including swallowing timing. The steps for extracting an index video 42 are described below. A user acquires a video file 41 of a swallowing test in the pharynx captured by the endoscope 13a. The video file 41 is transmitted to the index video creation unit 30. The video file 41 transmitted to the index video creation unit 30 is stored in the temporary storage area 31, and then the swallowing timing detection unit 32 analyzes the video file 41 to identify a swallowing frame image group 43, which is a frame image capturing a swallowing timing. The video file 41 from which the swallowing frame image group 43 is identified is transmitted to the index video extraction unit 33, and the swallowing frame image group 43 and a certain amount of frame images before and after the swallowing frame image group 43 are extracted as an index video 42. The index video 42 is transmitted to the swallowing classification unit 34, and a swallowing type classified by machine learning and an index number are assigned. The index video 42 is then transmitted to the display control unit 22, and a list of the swallowing frame images and automatic playback are performed on the display 14. After the index videos 42 are automatically played back and reviewed, the classification results of each index video 42 can be checked and editing such as additions and corrections can be performed. At this time, if the number of index videos 42 displayed is insufficient for the number of captured swallowing timings due to extraction failure or the like, or if the images are poorly captured, re-examination or manual extraction may be performed. If a sufficient number of index videos 42 are acquired, video information is edited for the information displayed in the video information display field 50. The index video 42 after video information editing is completed is stored in the storage memory 24 in the medical image processing device or in the database 12 of the connected device. In addition, it is preferable to delete the video file 41 stored in the temporary storage area 31 if it is not needed.

[0064] [Second embodiment] In the above embodiment, the swallowing timing is detected by image analysis of the acquired video file 41, but in this embodiment, the swallowing timing is judged and classified by adding voice recognition during swallowing. Extraction of the index video 42 in this embodiment will be described below. Note that the contents that overlap with the above embodiment will be omitted.

[0065] As a swallowing detection algorithm, swallowing in the oral phase is judged using voice. The user interface 15 connected to the medical image processing device 11 includes a microphone (not shown) for acquiring voice, and the voice waveform acquired from the microphone is input from the input receiving unit 23 to the medical image processing device 11 each time. The voice is acquired in conjunction with the imaging of the pharynx, and is sent to the index moving image creating unit 30 in the form of a moving image file 41 with the voice attached. The voice waveform is linked to the moving image file 41 as an audio signal, and is stored in the temporary storage area 31 of the index moving image creating unit 30, and then transmitted to the swallowing timing detecting unit 32.

[0066] As shown in Figure 17, in the second embodiment, the swallowing timing detection unit 32 realizes the functions of an examination video analysis unit 32a, a patient voice discrimination unit 32b, and a swallowing voice discrimination unit 32c, and determines whether or not there is a swallowing timing based on image analysis and voice discrimination.

[0067] The examination video analysis unit 32a performs swallowing detection processing by image analysis of the video file 41, similar to the processing performed by the swallowing timing detection unit 32 in the first embodiment. As a result, food F is recognized, and the video file 41 that is determined to have a swallowing timing is transmitted to the patient voice discrimination unit 32b. Note that it is preferable to store the video file 41 in which the swallowing timing is not detected in the storage memory 24. At that time, it is deleted from the temporary storage area 31.

[0068] The patient voice discrimination unit 32b analyzes the voice signal attached to the video file 41 determined to have a swallowing timing, and determines whether the voice is from the patient or from someone other than the patient. If the voice signal is determined to be from someone other than the patient, it is recorded together with the examination time. If the voice determined to be from someone other than the patient is a specific voice (e.g., a call such as "start examination") from a doctor or the like, the specific voice may be associated with a frame image of the video file 41 at the timing when the specific voice was emitted. If the voice signal is determined to have a voice from the patient during the swallowing timing, it is transmitted to the swallowing voice discrimination unit 32c. In addition, the frame image of the video file 41 linked to the determined voice may be tagged as "patient voice" or "non-patient voice".

[0069] The swallowing sound discrimination unit 32c judges whether the audio signal is a swallowing-related sound or a non-swallowing-related sound. The swallowing-related sound is a swallowing sound or an epiglottis opening and closing sound associated with swallowing, and the non-swallowing-related sound is a cough, a choking sound, a breathing sound, a vocalization, and the like. If the audio signal is a swallowing-related sound, a frame image of the video file 41 linked to the swallowing-related sound is tagged as "swallowing-related sound". Similarly, a frame image of the video file 41 linked to the non-swallowing-related sound is tagged as "non-swallowing-related sound". The video file 41 is transmitted to the index video extraction unit 33 regardless of the presence or absence of swallowing-related sound in the audio signal. It is preferable to judge whether the audio signal is a swallowing-related sound by calculating the probability that swallowing is in progress.

[0070] Even if the swallowing response cannot be determined from other responses by image analysis alone or the accuracy is low, the swallowing timing detection unit 32 configured as above can improve the accuracy of determining whether swallowing is in progress or not by using a voice signal to exclude other responses than swallowing, such as glottal closure and epiglottis opening and closing caused by coughing, which are not swallowing. ByIt is preferable that video files 41 that contain only "non-swallowing-related sounds" and are not tagged as "swallowing-related sounds" are extracted by the index video extractor 33 and then classified by the swallowing classifier 34 as being either aspiration or non-aspiration.

[0071] The index moving image extraction unit 33 sets frame images of a moving image file 41 tagged with "swallowing-related sounds" as a swallowing frame image group 43, which is the swallowing timing, and extracts the swallowing frame image group 43 and consecutive frame images of a certain number of seconds before and after it as an index moving image 42. For moving image files 41 not tagged with "swallowing-related sounds", index moving images 42 are extracted by only image analysis similar to that in the first embodiment.

[0072] The swallowing classification unit 34 performs classification by audio analysis on the index video 42 in addition to classification by image analysis. Classification is performed as to whether swallowing-related sounds and non-swallowing-related sounds are normal or abnormal (dysphagia) at the timing of their occurrence, and the classification result is added to the index video 42. Specifically, classification of swallowing-related sounds and non-swallowing-related sounds as normal or abnormal is determined after analyzing the number of swallowing-related sounds and non-swallowing-related sounds, the characteristics and length of the swallowing sounds, breathing sounds before and after swallowing, choking or coughing after swallowing, the intervals at which swallowing-related sounds are generated if multiple swallowing-related sounds are generated, and whether the epiglottis opening and closing sounds associated with swallowing are related to dysphagia. By combining with the classification result by image analysis, a more specific classification result or a more accurate classification result can be obtained.

[0073] Swallowing classification part 34 To After classification by the above method, the index video 42 is displayed and automatically played on the display 14 via the display control unit 22. It is preferable that the automatically played index video 42 also plays swallowing-related sounds in conjunction with the playback. It is also preferable that the video information display field 50 automatically displays information such as whether or not swallowing is normal in the information to be written.

[0074] [Third embodiment] In each of the above embodiments, the medical image processing device 11 obtains the moving image file 41 captured by the endoscope system 13 and extracts the index moving image 42, but in this embodiment, in addition to the extraction in each of the above embodiments, extraction is also performed from the moving image file 41 stored in the database 12. A review of the swallowing test in this embodiment will be described below. Note that content that overlaps with the above embodiments will be omitted.

[0075] Swallowing tests may be performed multiple times at intervals to track changes in pathological condition. Therefore, it is desirable to be able to compare the results of a swallowing test obtained with those of previous swallowing tests. The medical image processing device 11 receives video files 41 of previous swallowing tests from the database 12 using the image receiving unit 21, and extracts them using the index video creating unit 30.

[0076] As shown in FIG. 18, an index video 42 obtained and extracted from the endoscope system 13 and a past index video 47 obtained and extracted from the database 12 are displayed side by side on the display 14 to compare the swallowing behavior of food F.

[0077] When retrieving a specific video file 41 from the database 12, it is preferable to retrieve the file by performing a search on a search screen using the name of the type of swallowing, the patient's name, the date of shooting, etc., in order to confirm whether the swallowing in the retrieved index video 42 is normal or not.

[0078] The video obtained from the database 12 may be the video file 41 used to perform the extraction operation, or the extracted past index video 47 may be directly obtained and displayed on the display 14. Video synthesis such as the synthesized index video 46 shown in Fig. 11 may be performed using the index video 42 and the past index video 47. A synthesized index video 46 of the same swallowing type on different examination dates for the same patient is suitable for tracking pathological changes.

[0079] In each embodiment, the hardware structure of a processing unit that executes various processes, such as the central control unit 20, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically for executing various processes.

[0080] One processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same type or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

[0081] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit (Circuitry) in the form of a combination of circuit elements such as semiconductor elements, etc. Also, the hardware structure of the memory unit is a storage device such as an HDD (Hard Disc Drive) or an SSD (Solid State Drive). [Explanation of symbols]

[0082] 10 Medical image processing system 11 Medical image processing equipment 12 Database 13 Endoscope System 13a Endoscope 13b Endoscope tip 14 Display 15 User Interface 20 Central Control Unit 21 Image receiving unit 22 Display control unit 23 Input receiver 24 Storage Memory 30 Index video creation section 31 Temporary storage area 32 Swallowing timing detection unit 32a Inspection video analysis section 32b Patient voice discrimination unit 32c Swallowing sound discrimination unit 33 Index video extraction unit 34 Swallowing Classification Department 41 Video files 42 Index Video 42a Index Video 42b Index video 42c index video 42d index video 42e Index Video 43 Swallowing frame images 43a Swallowing frame image group 43b Swallowing frame image group 44 Swallowing Frame Image 45 Non-swallowing frame images 46 Composite Index Video 47 Past Index Videos 50 Video information display section 51 Play button 52 Rewind button 53 Fast forward button 54 Pause button 55 Slider 56 Seek Bar 57 Repeat play button 100 Example of frame image in Figure 12 101a Previous frame image in the top part of Figure 13 101b The rear frame image in the upper part of Figure 13 101c Example of the difference in the upper part of Figure 13 101d Previous frame image in the lower part of Figure 13 101e The rear frame image in the lower part of Figure 13 101f Example of the difference in the lower part of Figure 13 101g Image processing target area of ​​Fig. 13 102a Frame image in the top row of Figure 14 102b Example of edge amount in the upper part of Figure 14 102c The bottom frame image of Figure 14 102d Example of edge amount in the lower part of Fig. 14 102g Image processing target area of ​​Fig. 14 103a Frame image in the top row of Figure 15 103b The bottom frame image of Figure 15 103c Features 103g Image processing target area of ​​Fig. 15 Eg epiglottis F Food R position Rg glottic fissure Ps piriform pit

Claims

1. A processor is provided. The processor, Receives a video signal recorded by an endoscope during a swallowing test; The video signal is analyzed to determine whether or not there is a swallowing timing, and a frame image of the swallowing timing is treated as a swallowing frame image tagged with swallowing timing detection. A medical image processing system that extracts an index video including the swallowing frame image from the video signal.

2. The medical image processing system according to claim 1 , wherein the index video is composed of a swallowing frame image group including the swallowing frame image and a frame image group for a certain period of time subsequent to the swallowing frame image group.

3. The medical image processing system according to claim 2 , wherein the group of frame images for a certain period of time are non-swallowing frame images taken before and after the start and end of the group of swallowing frame images and are not tagged for swallowing timing detection.

4. the video signal includes a frame image to be analyzed; The processor, determining whether or not the swallowing timing is present by using any one of the calculation of the amount of blur of the frame images, the calculation of key points based on the frame images, or the difference in pixel values ​​between the frame images; 4. The medical image processing system according to claim 1, wherein the amount of blurring of the frame images is an edge amount indicating the magnitude of blurring between the frame images.

5. The processor, 5. The medical image processing system according to claim 1, further comprising: a step of analyzing the index video to identify a type of the swallowing test.

6. The processor, 5. The medical image processing system according to claim 1, wherein an index number for use in a video search is assigned to the index video.

7. The processor, 5. The medical image processing system according to claim 1, wherein the index moving image is automatically played back without a user operation when displayed on the screen.

8. The processor, 8. The medical image processing system according to claim 7, wherein a plurality of said index moving images are displayed on a display in a list form, and simultaneous or continuous playback is automatically performed.

9. The processor, The medical image processing system according to claim 1 , wherein at least one of the type of the swallowing test and an index number is displayed when the index moving image is displayed on a screen.

10. The processor, 2. The medical image processing system according to claim 1, wherein a plurality of said index moving images are joined together to generate a composite index moving image that enables said index moving images to be played back continuously.

11. The processor, 2. The medical image processing system according to claim 1, wherein the swallowing timing is determined by voice recognition during swallowing.

12. 1. A method of operating a medical imaging system including a processor, comprising: The processor, receiving a video signal recording a swallowing test by an endoscope; A step of analyzing the video signal to determine whether or not there is a swallowing timing, and treating the frame image of the swallowing timing as a swallowing frame image tagged with swallowing timing detection; and extracting an index video from the video signal, the index video including the swallowing frame images.

Images