Medical support device, medical support method, and medical support program

Abstract

To reduce the workload for the operator or diagnostician during colposcopy. [Solution] A medical support device for performing colposcopy, comprising a control unit that, based on a plurality of images obtained by continuous shooting of a target area to be continuously subjected to acetic acid processing, identifies the timing at which a predetermined change related to acetic acid processing occurs in the target area or the image taken at that timing, and controls at least one of the following based on the identified timing or identified image: shooting condition control in continuous shooting, image recording control in continuous shooting, and display control of the image recorded by continuous shooting.

Description

Technical Field

[0001] The present invention relates to a medical support device, a medical support method, and a medical support program.

Background Art

[0002] As one of the examinations for cervical cancer, colposcopy is performed to observe the cervix using a colposcope (vaginal magnifying mirror). In colposcopy, by applying acetic acid to the cervix, if there is a lesion, the surface changes color, and the lesion site can be observed. For example, Patent Document 1 discloses a method of taking pictures before and after applying acetic acid processing to a lesion and detecting the lesion site based on the taken images.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in addition to photographing the lesion, no consideration has been given to reducing the labor of the operator or diagnostician who performs various operations such as applying acetic acid, observing the lesion, determining the necessity of biopsy, collecting cells or tissues for biopsy, recording the working conditions, and recording the diagnosis of the lesion. This was the same not only when applying acetic acid to the cervix but also when applying acetic acid or other chemical solutions to other disease sites such as the otolaryngological region or the digestive tract to make it easier to observe the state of the lesion.

[0005] The present invention has been made to solve the above problems, and an object thereof is to reduce the labor of the operator or diagnostician who observes and diagnoses a lesion.

Means for Solving the Problems

[0006] To achieve the above objective, the medical support device according to the present invention is a medical support device that assists in performing colposcopy, and includes a control unit that, based on a plurality of images obtained by continuous shooting of a target area to be continuously subjected to acetic acid processing, identifies the timing at which a predetermined change related to acetic acid processing occurs in the target area or the image taken at that timing, and controls at least one of the following based on the identified timing or the identified image: control of the shooting conditions in the continuous shooting, control of image recording in the continuous shooting, and control of the display of the image recorded by the continuous shooting. [Effects of the Invention]

[0007] According to the present invention, the workload of the operator or diagnostician in colposcopy can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an overview of the overall configuration of a medical support device according to the first embodiment of the present invention. [Figure 2] This flowchart shows the control procedure for the acetic acid processing mode according to the first embodiment of the present invention. [Figure 3] (a) shows the display unit of the medical support device, and (b) shows the display screen in acetic acid processing mode. [Figure 4] This flowchart shows the control procedure for the acetic acid processing mode according to the first embodiment of the present invention. [Figure 5] This flowchart shows the control procedure for the acetic acid processing regeneration-dedicated mode according to the first embodiment of the present invention. [Figure 6] This flowchart shows the control procedure for the biopsy mode according to the first embodiment of the present invention. [Figure 7] This is a diagram showing the display screen in biopsy mode. [Figure 8] This is a flowchart showing the image extraction process according to the second embodiment of the present invention. [Figure 9] This is a flowchart showing the image extraction process according to the second embodiment of the present invention. [Figure 10] This is a flowchart showing the image extraction process according to the second embodiment of the present invention. [Modes for carrying out the invention]

[0009] (First embodiment) The medical support device according to the embodiment of the present invention is applicable to a colposcopy camera, which is an imaging device for examining the vagina or cervix, and more generally, a close-up camera, which is an imaging device that approaches a subject to photograph its details. It may also be a system combining an imaging device such as a colposcopy camera with a viewer device. The viewer device may be an information processing device such as a personal computer. Here, a colposcopy camera will be used as an example.

[0010] As shown in Figure 1, the medical support device 1 comprises a CPU (Central Processing Unit) 2, RAM (Random Access Memory) 3, storage unit 4, imaging unit 5, light-emitting unit 6, display unit 7, notification unit 8, operation unit 9, voice input unit 10, and communication unit 11, and each of these units is connected by a bus.

[0011] CPU2 is a processor that acts as a control unit, controlling the operation of the medical support device 1 by reading and executing the program 12 stored in the memory unit 4 and performing various arithmetic operations. The medical support device 1 may have multiple processors (for example, multiple CPU2s), and the multiple processes performed by the CPU2 in this embodiment may be performed by these multiple processors. When imaging the cervix, CPU2 causes the light-emitting unit 6 to emit light according to settings and operations by the operator or diagnostician (hereinafter referred to as "operator, etc."), and causes the imaging unit 5 to take an image according to the operator, etc.'s instructions to take an image, thereby generating image data of a medical image of the cervix. RAM3 functions as a work area for CPU2 and temporarily stores the program 12 and data. The imaging unit 5 includes an optical system for forming an image of incident light, an image sensor for detecting the incident light formed by the optical system, and an image processing unit that generates image data based on the detection signal output from the image sensor. The optical system can be configured, for example, with mirrors and lens groups. For example, a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) can be used as the image sensor. The image processing unit amplifies the detection signal from the image sensor and performs A / D conversion to generate image data of the captured subject. The image processing unit may also perform various image processing operations on the generated image data.

[0012] The light-emitting unit 6 emits illumination light in the direction of imaging by the imaging unit 5 to illuminate the subject. As described above, the light-emitting unit 6 is equipped with a normal light, a green light, and a polarizing light. The on / off switching of the light emission by the light-emitting unit 6 is controlled by a control signal transmitted from the CPU 2. The display unit 7 is equipped with a display device such as a liquid crystal display. The display unit 7 displays the captured image in real time as a live view image on the display screen of the display device according to the image data and control signals transmitted from the CPU 2, or displays captured images in playback, or displays various operation icons. The notification unit 8 is equipped with a speaker that generates a notification sound. As will be described later, the notification unit 8 notifies the operator, etc., by generating a detection sound when a foreign object is detected in the image displayed by the display unit 7, or when there is a change in color tone in the image, etc. The operation unit 9 has a touch panel superimposed on the display screen and operation buttons provided on the camera housing. The operation unit 9 detects contact of fingers, etc., with the touch panel and operation of the operation buttons, and outputs an operation signal to the CPU 2 according to the detection result. The voice input unit 10 is equipped with a microphone for inputting voice. For example, when a surgeon or other person speaks a specific word and voice is input to the microphone, it outputs an operation signal corresponding to that word to the CPU 2. The communication unit 11 is equipped with a communication module such as a LAN and communicates data with a medical image management device, etc. via a network.

[0013] Medical support device 1 has the following functions to assist the operator during colposcopy: an acetic acid processing mode for applying acetic acid to the cervix to observe lesions; an acetic acid processing dedicated playback mode for assisting the operator in extracting images showing the strongest lesion areas during acetic acid processing; and a biopsy mode for assisting the operator in biopsy, which involves taking tissue samples from lesion areas detected by acetic acid processing. Each of these modes will be explained in Figure 2 and subsequent figures.

[0014] (Acetic acid processing mode) FIG. 2 is a flowchart showing the control procedure executed by the CPU 2 in the acetic acid processing mode. When the medical support device 1 is installed so that the imaging unit 5 can photograph the target site of the patient who is the subject, first, an operator or the like sets the camera body to the "acetic acid processing mode" (step S101). Here, the "acetic acid processing mode" is turned on by pressing a button on the camera body side or touching a screen on the display device, and a screen in the acetic acid processing mode is displayed.

[0015] When the "acetic acid processing mode" is entered, the imaging unit 5 performs shooting with an automatic shutter (step S102). At the time of shooting, the light emitting unit 6 drives the normal light, green light, and polarized light in order, and irradiates the target site of the patient who is the subject with the normal light, green light, and polarized light as illumination light. The imaging unit 5 performs shooting every time each illumination light is irradiated. Here, shooting means selectively recording an image taken at the timing of an automatic shutter or the like from a live view image that continuously displays shooting images in real time. Thereby, an image of the target site of the patient before acetic acid processing is taken.

[0016] After shooting is performed, an operator or the like applies acetic acid by pressing a cotton ball soaked with an acetic acid solution having a concentration of, for example, 3 to 5% against the target site of the patient for 10 to 15 seconds. When the application of acetic acid is completed, support for observing the target site after acetic acid processing is started by a start button operated by an operator or the like. The start button is provided on the camera body side, and the CPU 2 determines whether or not the start button has been pressed (step S103). When the CPU 2 determines that the start button has been pressed (step S103: Yes), it starts supporting the observation of the target site after acetic acid processing from step S110 and subsequent steps. Note that the start button may be displayed on the display screen of the display unit 7, and support for observing the target site after acetic acid processing is started by touching the start button displayed on the screen.

[0017] On the other hand, when the start button is not pressed and the CPU 2 determines that the start button has not been pressed (step S103: No), it then determines whether there has been a voice start input (step S104). When an operator or the like utters a specific word, for example, "start", the voice input unit 10 detects this and outputs an operation signal corresponding to the detection result to the CPU 2. When the CPU 2 receives the output from the voice input unit 10 and determines that there has been a voice start input (step S104: Yes), it starts assisting the observation of the target site after acetic acid processing from step S110 onwards. As a result, the operator or the like can start assisting the observation of the target site after acetic acid processing at the timing determined by themselves without touching the camera.

[0018] On the other hand, when the word "start" is not uttered and the CPU 2 determines that there has been no voice start input (step S104: No), it then starts assisting the observation of the target site after acetic acid processing from step S110 onwards based on detecting the end of the acetic acid processing from the image of the target site input from the imaging unit 5. Here, the image input from the imaging unit 5 is a live view image obtained by continuously shooting the image input from the imaging unit 5. First, image processing is performed on the image of the target site input from the imaging unit 5 to determine whether a foreign object other than the target site is detected (step S105). The image processing unit of the imaging unit 5 performs color extraction from the image of the target site, performs edge detection on the extracted color region, and extracts the contour shape, which is the geometric information of the color region, to detect a foreign object. Here, a foreign object is an instrument such as forceps, tweezers, pliers, cotton balls, syringes, etc. other than the cervix, which is the target site. If the foreign object is reflected in the image, it indicates that acetic acid processing of the target site is in progress. On the other hand, if the foreign object is not reflected in the image, it indicates that it is before or after the acetic acid processing of the target site.

[0019] In step S105, if it is determined that a foreign object has been detected (step S105: Yes), the area where the foreign object was detected is marked, and the marked area is displayed on the display device screen with a dotted line or the like (step S106). Here, since a foreign object has been detected, the acetic acid processing treatment is being performed on the target area. When a foreign object is detected, the notification unit 8 emits a notification sound, such as a "beep," to inform the operator or other person of the detection of a foreign object.

[0020] In contrast, if it is determined that no foreign object has been detected in step S105 (step S105: No), it is determined that the acetic acid processing on the target area has not yet been performed, and the process returns to step S103, repeating steps S103 to S105 until a foreign object is detected. However, if the start button is pressed or a voice start input is received during this time, the process proceeds to step S110 and beyond, which assists in starting the observation of the target area after acetic acid processing. This allows the operator to start the process assisting in the observation of the target area after acetic acid processing even if no foreign object has been detected, once the acetic acid processing is complete.

[0021] Returning to step S106, once the detected foreign object is marked, the CPU2 determines whether the start button has been pressed, similar to step S103 (step S107). At this point, the operator can press the start button at their own determined timing while checking the marked area on the display screen. If the CPU2 determines that the start button has been pressed (step S107: Yes), it begins assisting with the observation of the target area after acetic acid processing, starting from step S110.

[0022] In contrast, if the start button is not pressed and the CPU2 does not determine that the start button has been pressed (step S107: No), then, similar to step S104, it determines whether or not there has been an audio start input (step S108). If the CPU2 receives the output from the audio input unit 10 and determines that there has been an audio start input (step S108: Yes), it starts assisting with the observation of the target area after acetic acid processing from step S110 onward. This allows the operator to start assisting with the observation of the target area after acetic acid processing at a timing of their choosing without touching the camera.

[0023] In response, if the CPU 2 does not determine that there has been an audio start input (step S108: No), it determines whether or not foreign objects other than the target area are detected in the image of the target area input from the imaging unit 5 (step S109). If it is determined that no foreign objects are detected (step S109: No), that is, when the marking on the display device screen disappears and the foreign object disappears from the screen, it determines that the acetic acid processing on the target area is complete and starts supporting the observation of the target area after acetic acid processing from step S110 onwards. Also, when no foreign objects are detected, the notification unit 8 emits a notification sound, such as a "beep," to indicate that no foreign objects are detected and to inform the operator that the foreign object has disappeared from the screen and the acetic acid processing is complete.

[0024] In contrast, if the CPU2 determines in step S109 that a foreign object has been detected (step S109: Yes), the process returns to step S107, and steps S107 to S109 are repeated until the CPU2 detects that the foreign object has disappeared from the screen. However, if the start button is pressed or a start input is received via voice during this time, the process proceeds to step S110 and beyond, which is the process for starting support for observing the target area after acetic acid processing. This ensures that even if the acetic acid processing is completed without the disappearance of the foreign object being detected by mistake, the operator can start the support process for observing the target area after acetic acid processing.

[0025] When acetic acid is applied to the cervix, if there is a lesion, the epithelium will turn whiter than the surrounding normal epithelium. In addition, a yellowish tint may be observed as a change. Hereinafter, the above reaction in which the cervix turns white (yellow) will be referred to as the white reaction. In step S110, the imaging unit 5 takes an image with an automatic shutter. When taking an image, the light-emitting unit 6, as part of the imaging condition control, does not drive the green light, but instead drives the normal light and polarized light in sequence to irradiate the target area of ​​the patient with normal light and polarized light. The imaging unit 5 takes an image each time each illumination light is irradiated. In this way, the imaging unit 5 takes an image of the target area before the white reaction.

[0026] When an image of the target area before the white reaction is captured, the display unit 7 switches the display screen of the display device from a single screen display showing only the live view image input from the imaging unit 5 to a two-screen display that adds the captured image of the target area before the white reaction by splitting the screen into two (step S111). Here, Figure 3 shows an example of the display device screen with two screens. Figure 3(a) shows the screen 20 of the medical support device 1, and Figure 3(b) shows the state with two screens displayed. As shown in Figure 3(b), the screen is divided into left and right halves, with the image of the target area before the white reaction automatically captured on the left screen 21, and the live view image during the white reaction displayed on the right screen 22. By displaying the image before the white reaction and the live view image during the white reaction side by side, the operator can check the target area before the reaction and compare and confirm the changes in real time.

[0027] When the display switches to dual-screen mode, CPU2 begins measuring the elapsed time after acetic acid processing (step S112). Once time measurement begins, the elapsed time is displayed on screen 20 of the display device. This allows the operator to check the elapsed time while observing the target area.

[0028] When the measurement of time since acetic acid processing begins, in step S113 of Figure 4, the CPU2 determines whether the operator or other person has performed a stop operation to end the measurement. If the stop operation has not been performed (step S113: No), the CPU2 determines whether or not the disappearance of the white reaction has been detected (step S114). If the disappearance of the white reaction is not detected (step S114: No), the CPU2 determines whether or not it is time for automatic imaging (step S115). If it is not time for automatic imaging (step S115: No), the CPU2 determines whether or not there has been a change in the white reaction, that is, whether or not the epithelium of the target area has changed to white (yellow) (step S116). The change in the white reaction is determined by the increase or decrease in the area of ​​white (yellow) and changes in color tone such as intensity. The result of the determination in step S116 is used as the timing for imaging the target area, as will be described later. Here, the change in the white reaction is determined by whether the difference between the state of the white reaction in the current assessment and the state of the white reaction in the previous assessment is greater than or equal to a predetermined value, and automatic imaging may be performed each time the difference exceeds the predetermined value. Alternatively, the change in the white reaction may be determined by whether the amount of change in the state of the white reaction in the current assessment compared to the state before the white reaction occurred is greater than or equal to a predetermined value, and automatic imaging may be performed at predetermined intervals as long as the difference exceeds the predetermined value. Furthermore, these judgment methods, the difference in change, the amount of change, predetermined values, predetermined times, and other conditions may be set by the operator or other personnel. Immediately after the start of measurement, stop operations, detection of the disappearance of the white reaction, and automatic imaging are not usually performed, so the above judgment is made. Details of steps S113 to S115 will be described later.

[0029] During colposcopy, after applying acetic acid to the target area, the time it takes for the white reaction to appear and disappear is observed. Here, the time of appearance and disappearance differs depending on the lesion. In mild cases, the white reaction appears and disappears quickly. In contrast, in severe cases, it takes time for the white reaction to appear and disappear. The period 30 to 60 seconds after acetic acid application is when the findings are clearest and is suitable for observation, but the more malignant the lesion, the longer the findings persist, and observation may take more than 3 minutes. If the white reaction does not appear in step S116 (step S116: No), return to step S113 and repeat the process from steps S113 to S116 until the white reaction appears.

[0030] In step S112, when the measurement of the elapsed time after acetic acid processing begins and a white (yellow) change starts to appear in the epithelium of the target area, and the appearance of a white reaction is detected (step S116: Yes), the CPU2 calculates the time required from the start of acetic acid processing until the appearance of the first white reaction and records it in the memory unit 4 (step S117). Next, the notification unit 8 notifies the operator, etc., that a white reaction has appeared by emitting an alarm sound (step S118). Subsequently, in order to photograph the area where the white reaction has appeared, the CPU2 instructs the imaging unit 5 to perform focus control to focus on the detected area as part of the shooting condition control, and the imaging unit 5 focuses on the area where the reaction occurred (step S119). Once the area where the white reaction has appeared is in focus, the CPU2 instructs the imaging unit 5 to take an image of the area where the white reaction has appeared as part of the image recording control. The imaging unit 5 takes an image with an automatic shutter (step S120). When taking an image, the CPU2 instructs the light-emitting unit 6 to switch the illumination light as part of the shooting condition control. The light-emitting unit 6 drives the normal light and polarized light in sequence to irradiate the target area with normal light and polarized light, and the imaging unit 5 takes a picture each time each illumination light is irradiated. In this way, the imaging unit 5 captures an image of the target area immediately after the white reaction. Once the picture is taken, the imaging unit 5 records the time of the capture (step S121).

[0031] Once the shooting and recording of the shooting time are complete, it is determined whether the change in color tone due to the white reaction is at its maximum, that is, whether the white reaction has reached its peak and the color tone of the area where the white reaction is appearing is the strongest and clearest (step S122). At the beginning of the appearance of the white reaction, the area of ​​the white reaction gradually expands and the color tone also changes, so it is not determined that the white reaction has reached its peak (step S122: No), and the process returns to step S113, and the processing from step S113 onwards is executed again. If the white reaction persists, the operator or other person does not usually perform a stop operation in step S113, and the disappearance of the white reaction is not detected in step S114. If the automatic shooting timing is reached in step S115 (step S115: Yes), the imaging unit 5 automatically takes a picture of the target area (step S120). The automatic shooting performed here is performed at predetermined intervals according to the elapsed time since the acetic acid treatment in step S112. For example, after acetic acid treatment, a predetermined number of shots are taken within the time from when the white reaction begins to appear until it disappears. As mentioned above, the period from 30 to 60 seconds after acetic acid application is the time when the findings are clearest and is suitable for observation. Therefore, from 30 to 90 seconds after acetic acid application, including this period, images will be taken every 10 seconds. Furthermore, since observation may take more than 3 minutes, images will be taken every 30 seconds from 90 to 300 seconds, including this time. This ensures that images are taken at the above-mentioned pre-set time intervals to account for cases where the appearance, color change, or fading of the white reaction cannot be detected and automatic imaging fails to record, thus preventing any recording omissions.

[0032] If the white reaction persists and the automatic shooting timing has not been reached in step S115, then in step S116, the white reaction has changed, so the imaging unit 5 focuses on the area with the white reaction and automatic shooting is performed, and the shooting time is recorded in the memory unit 4 (steps S119, S120, S121). Note that since the time until the first appearance of the white reaction has already been recorded, steps S117 and S118 are not performed. In this way, as long as the white reaction persists, the target area is repeatedly photographed.

[0033] As the white reaction in the target area progresses and reaches its peak, the white reaction begins to subside. In step S122, the CPU2 detects this change and determines that the intensity of the color in the previously observed white reaction area is at its maximum (step S122: Yes), and records the location and shape of the area where the white reaction is appearing in the storage unit 4 (step S123). This ensures that a clear image of the peak of the white reaction is recorded without being missed.

[0034] As mentioned above, once the white reaction has passed its peak, it begins to subside. However, as long as the white reaction persists, the target area is repeatedly photographed. When the white reaction becomes smaller and the operator determines that it has subsided and wishes to end the observation, they perform a stop operation. The stop operation is performed by operating the stop button on the camera body or by touching the stop button displayed on the display screen of the display unit 7. Alternatively, the operator may generate a voice command to stop, which the voice input unit 10 detects and generates an operation signal to execute the stop operation. When the CPU 2 determines that the operator has performed a stop operation (step S113: Yes), it records the time from the appearance to the subsidence of the white reaction (step S124). If the operator does not perform a stop operation, the CPU 2 detects the subsidence of the white reaction (step S114: Yes), terminates the white reaction photography process, and records the time from the appearance to the subsidence of the white reaction (step S124). Once the recording of the time from the appearance to the disappearance of the white reaction is complete, the measurement of the elapsed time after acetic acid processing is terminated (step S125), and the display screen of the display device is switched from a two-screen display of the image of the target area before the white reaction and the live view image input to the imaging unit 5 to a single-screen display of only the live view image (step S126). When the screen display is returned to its original state, the acetic acid processing mode is automatically turned OFF (step S127), and the acetic acid processing mode is terminated. As described above, the acetic acid processing mode assists in reducing the workload of the operator and others, and since the necessary images are saved at the appropriate timing, the amount of memory required for storage can be reduced.

[0035] (Regeneration mode for acetic acid processing only) As described above, multiple images are captured of the target area using the acetic acid processing mode. Once the capture is complete, the captured images are played back for review. In the normal playback mode, playback is performed in the order in which the images were captured. In this embodiment, in addition to this, a playback mode specifically for acetic acid processing is provided as a playback function that allows for easy identification of the most prominent lesion area from among the multiple images captured in the acetic acid processing mode. The playback mode specifically for acetic acid processing is a function that plays back according to the intensity of the color tone of the white reaction area, assisting the operator in extracting the image showing the most prominent lesion area. Next, the playback mode specifically for acetic acid processing will be explained with reference to the flowchart in Figure 5.

[0036] In image playback, the operator first switches to "Acetic Acid Processing Dedicated Playback Mode" by operating the camera body (Step S201). Here, "Acetic Acid Processing Dedicated Playback Mode" is turned ON by pressing a button on the camera body or touching the screen on the display device, and the screen for Acetic Acid Processing Dedicated Playback Mode is displayed.

[0037] When the system switches to the "Acetic Acid Processing Dedicated Playback Mode," a list of images taken in the most recent acetic acid processing mode is displayed on the display device screen (step S202). Here, the list of images is displayed in order of the time they were taken. Next, the CPU 2 instructs the display unit 7 to display according to the change in the white reaction. The display unit 7 rearranges the images, which are arranged in order of the time they were taken, in order of the intensity of the white reaction portion, and plays them back in the rearranged order (step S203). Here, the images that are rearranged are those that were automatically taken in response to the detection of a change in the white reaction in step S116 of Figure 4 of the acetic acid processing mode. Note that multiple images that were automatically taken at a preset time interval may also be automatically rearranged and displayed in order of the intensity of the white reaction portion. However, these images are saved in a separate folder from the images that were automatically taken in response to the detection of a change in the white reaction.

[0038] During image playback, the display unit 7 additionally displays the required time for acetic acid processing on the screen (step S204). The required time for acetic acid processing is the time required from the application of acetic acid to the target area until the white reaction appears, and the time recorded in step S117 of Figure 4 in acetic acid processing mode is used. The display unit 7 also additionally displays the time required from the appearance to the disappearance of the white reaction on the screen (step S205). For the time required from the appearance to the disappearance of the white reaction, the time recorded in step S124 of Figure 4 in acetic acid processing mode is used. Since the appearance time and disappearance time differ depending on the lesion, these required times serve as a reference for extracting the strongest lesion site and as a basis for judging the findings. The operator confirms the above-mentioned playback image and presses the "Confirm" button for "Strongest Lesion" on the image that is most appropriate as the definitive image of the strongest lesion site, and the image of the strongest lesion site is confirmed (step S206). This reduces the time required for operators to confirm the location of the most severe lesion (biopsy site) during acetic acid processing procedures and when recording after biopsy.

[0039] (Biopsy mode) As described above, once the image of the most severe lesion is extracted using the acetic acid processing dedicated regeneration mode, a targeted biopsy is performed, selecting the most severe lesion as the biopsy site and collecting tissue from that area. At this time, by using the biopsy mode function, the operator can perform the targeted biopsy while comparing and confirming with the extracted image of the most severe lesion. Next, the biopsy mode will be explained with reference to the flowchart in Figure 6.

[0040] When performing a biopsy, the operator first switches the camera to "biopsy mode" (step S301). Here, "biopsy mode" is turned ON by pressing a button on the camera body or touching the screen on the display device, and the biopsy mode screen is displayed.

[0041] When the system switches to "biopsy mode," the display unit 7 switches the display screen of the display device from a single screen displaying only the live view image input from the imaging unit 5 to a two-screen display that splits the screen into two and adds the image of the most severe lesion confirmed in the acetic acid processing dedicated playback mode (step S302). Here, Figure 7 shows an example of the display device screen with two screens. As shown in Figure 7, the screen 30 is split into two, left and right, with the image of the most severe lesion confirmed in the acetic acid processing dedicated playback mode displayed on the left screen 31, and the live view image during the biopsy displayed on the right screen 32. By displaying the image of the most severe lesion and the live view image during the biopsy side by side, the operator can perform the biopsy while confirming the most severe lesion.

[0042] The image processing unit of the imaging unit 5 performs image processing on the image of the biopsy site input from the imaging unit 5 and determines whether or not foreign objects other than the biopsy site have been detected (step S303). Here, foreign objects refer to instruments used in biopsy other than the target site, the cervix, such as forceps, tweezers, cotton balls, syringes, etc. If such foreign objects are captured in the image, it indicates that a biopsy is being performed. Therefore, if it is determined that foreign objects other than the biopsy site have been detected (step S303: Yes), the imaging unit 5 takes an image of the biopsy site during the biopsy using the automatic shutter (step S304). When taking an image, the CPU 2 instructs the light-emitting unit 6 to switch the illumination light as part of the shooting condition control. The light-emitting unit 6 drives the normal light and polarized light in sequence to irradiate the target site with normal light and polarized light, and the imaging unit 5 takes an image each time each illumination light is irradiated. Conversely, if the foreign objects are not captured in the image, it indicates that the biopsy has not yet been performed. Therefore, if it is determined that no foreign body other than the biopsy site has been detected (Step S303: No), the process in Step S303 is repeated until a foreign body is detected and it is determined that the biopsy is in progress. When automatic imaging is performed in Step S304, the process of determining whether or not a foreign body has been detected is performed again (Step S305). If no foreign body is detected here (Step S305: No), it is determined that the instruments used for the biopsy have been removed and the biopsy is complete, and the imaging unit 5 takes an image with the automatic shutter to record an image of the biopsy site after the completion of the biopsy (Step S306). On the other hand, if the foreign body is still detected (Step S305: Yes), it indicates that the biopsy is continuing, and the process in Step S305 is repeated until no foreign body is detected and it is determined that the biopsy is complete. When automatic imaging is completed in Step S306, the biopsy mode is automatically turned OFF (Step S307), and the biopsy mode ends. Note that the biopsy mode may also be turned OFF by the operator or other operator using buttons, etc. As described above, by using the biopsy mode, the biopsy site is automatically photographed at the appropriate time, allowing the operator to save an accurate record of the biopsy site, and the captured record is useful for validating the biopsy site from which the tissue was taken.Furthermore, the operator, wearing infection-prevention gloves for contact with the living body, can automatically take images by detecting the instruments used in the biopsy, without having to touch the camera during the procedure.

[0043] (Second embodiment) In the above embodiment, an acetic acid processing mode is provided, and during observation of acetic acid processing, the medical support device 1 measures the time required from the appearance of the white reaction after acetic acid processing until the white reaction disappears, displays and records this information on the display device screen, and automatically takes multiple still images of the target area, including the most severe lesion, from the appearance to the disappearance of the white reaction, and records them along with the time of capture. In contrast, in this embodiment, a video of the target area is taken during observation of acetic acid processing, and after the observation is completed, an image extraction process is performed to extract and record multiple images from the video showing the above-mentioned time required and from the appearance to the disappearance of the white reaction.

[0044] Figures 8 to 10 are flowcharts showing the control procedure by which the CPU2 executes the image extraction process. Figure 8 shows the process of detecting acetic acid processing, steps S403 to S408 show the process of detecting the start of acetic acid application, and steps S409 to S414 show the process of detecting the end of acetic acid application.

[0045] First, a video of the target area captured by the medical support device 1 during observation of the target area before and after acetic acid processing is loaded (step S401). Once the video is loaded, the count value fc of the foreign object counter, which increments for each frame in which a foreign object is detected in the image, and the count value nfc of the no-foreign-object counter, which increments for each frame in which no foreign object is detected in the image, are both reset to 0 (step S402). Next, a process to detect the start of acetic acid application is executed. First, one frame of the video is advanced (step S403). Subsequently, image analysis is performed on the frame image (step S404), color is extracted from the image, edge detection is performed on the extracted color region, and the contour shape, which is the geometric information of the color region, is extracted to detect the foreign object. The CPU 2 determines whether or not a foreign object has been detected based on the image analysis (step S405). If no foreign object is detected (step S405: No), the count value fc of the foreign object counter is cleared to 0 (step S406), and the process proceeds to step S403, where the next frame is image analyzed to detect foreign objects and determine whether or not a foreign object has been detected. The process from steps S403 to S406 is then repeated until a foreign object is detected. If a foreign object is detected, CPU2 increments the count value fc of the foreign object counter by +1 (step S407), and then determines whether or not the count value fc has reached the set value "60" (step S408). Here, "60" is the number of frames, representing 1 second if the frame rate is 60fps, and 2 seconds if the frame rate is 30fps. In other words, it represents the elapsed time, and the process determines whether or not the time during which foreign objects are continuously detected has reached the set time. If the count value fc has not reached 60 (step S408: No), the process returns to step S403, and the process from step S403 onward is repeated. In step S408, if the count value fc = 60 (step S408: Yes), it is determined that foreign matter has been continuously detected within the set time, that is, that the process of applying acetic acid to the target area using the instruments used for acetic acid processing has begun. The set value can be arbitrarily set by the operator, etc., taking into account the frame rate of the video, but it is necessary to set a value that corresponds to the time during which foreign matter is continuously detected to the extent that it is recognized as the start of acetic acid application.

[0046] When the start of acetic acid application is detected in step S408, the CPU2 then executes the process of detecting the end of acetic acid application in steps S409 to S414. In step S409, the CPU2 advances the video frame by frame, performs image analysis on the image (step S410), and determines whether foreign matter is still being detected based on the image analysis (step S411). If foreign matter is continuously detected (step S411: Yes), the count value nfc of the foreign matter-free counter is cleared to 0 (step S412), and the process returns to step S409, and steps S409 to S412 are repeated until no foreign matter is detected. When no foreign matter is detected, the count value nfc of the foreign matter-free counter is increased by +1 (step S413), and it is determined whether the count value nfc has reached the set value "60" (step S414). As before, this set value is the number of frames and indicates the elapsed time, and it is determined whether foreign matter has not been continuously detected within the time set here. If the count value nfc=60 has not been reached (step S414: No), the process returns to step S409 and the processing from step S409 onward is repeated. If the count value nfc=60 (step S414: Yes), it is determined that no foreign matter has been detected continuously within the set time, i.e., the process of applying acetic acid to the target area using the instruments used for acetic acid processing has been completed. The set value can be arbitrarily set by the operator, etc., taking into consideration the frame rate of the video, but it is necessary to set a value that corresponds to the time during which no foreign matter is detected continuously enough to be considered as the completion of acetic acid application.

[0047] When it is determined that the acetic acid coating is complete, CPU2 extracts the still image i(0) that was being analyzed at that time from the video (step S415), displays and records it, and also records the playback time t0 of the video up to the still image i(0) (step S416).

[0048] Figure 9 shows the image extraction process from the appearance of the white reaction to the most severe lesion. Steps S417 to S421 show the image extraction process at the time of the appearance of the white reaction, and steps S423 to S429 show the image extraction process from the appearance of the white reaction to the most severe lesion. In step S417, one frame of the video is advanced, and in step S418, image analysis is performed on the frame image. Based on the changes in the acetic acid processed area detected by the image analysis, CPU2 determines whether the increase in the white (yellow) area in that area is above a certain level (step S419). The white reaction is detected by the change in the color density and area of ​​the white (yellow) area. Here, the change in the white (yellow) area is determined by whether the amount of increase in the white (yellow) area relative to the state of the acetic acid processed area in the image extracted in step S415 at the end of acetic acid processing is above a predetermined value. Alternatively, it may be determined by whether the amount of increase in the white (yellow) area relative to the image one or several frames earlier is above a predetermined value. If the white (yellow) area increases, that is, if the white (yellow) area becomes darker and the area of ​​white (yellow) increases, and the amount of increase is not above a predetermined level (Step S419: No), the process returns to Step S417, and the processing from Step S417 onward is executed, and the video playback continues until the increase in the white (yellow) area reaches a predetermined level or higher. If the amount of increase in the white (yellow) area is above a predetermined level or higher (Step S419: Yes), it is determined that a white reaction has appeared, and the frame image in question is extracted from the video as a still image i(1) at the time the white reaction appeared (Step S420), displayed and recorded, and the playback time t1 of the video up to the still image i(1) is recorded (Step S421).

[0049] When a still image i(1) at the time of the appearance of the white reaction is extracted, the variable n is set to 2. Here, the variable n represents the nth still image i(n) extracted after the appearance of the white reaction. Subsequently, image extraction processing is performed from the time of the appearance of the white reaction up to the strongest lesion. In step S423, one frame of the video is advanced, and in step S424, image analysis is performed on the frame image. Based on the changes in the white reaction detected by the image analysis, the CPU2 determines whether or not it has detected a decline in the white reaction, which is changing from an increasing direction to a decreasing direction (step S425). If the white reaction has not changed in the decreasing direction (step S425: No), it determines whether or not the increase in the white reaction is above a certain level (step S426). Here, whether or not the increase in the white reaction is above a certain level is determined by whether or not the amount of increase in the white (yellow) area relative to the still image i(1) at the time of the appearance of the white reaction is above a predetermined value. Alternatively, it may be determined by whether or not the amount of increase in the white (yellow) area relative to the image one or several frames earlier is above a predetermined value. If the increase in the white reaction is not above a predetermined level (step S426: No), the process returns to step S423, and the processing from step S423 onwards is executed, and the video is played back frame by frame until the increase in the white reaction reaches a certain level or higher. If the increase in the white reaction is above a predetermined level or higher (step S426: Yes), it is determined that the white reaction is continuing, and the frame image is extracted from the video as a still image i(2) after the appearance of the white reaction (step S427), displayed, and recorded. Once the still image i(2) is extracted, the variable n is increased by 1 and changed to n+1. Here, n=2+1=3. When the variable n is changed, the process returns to step S423, and the processing from step S423 onwards is executed. If the increase in the white reaction is above a certain level or higher, the still image i(n) is extracted, displayed, and recorded, the variable n is changed to n=n+1, and the processing from step S423 onwards proceeds in the same manner. During the above repeated processing, if the increase in the change in the white reaction falls below a predetermined level and the white reaction changes in the direction of decreasing, and a decline in the white reaction is detected (Step S425: Yes), it is determined that the strongest lesion has been detected, and the image from the frame immediately before the decline is extracted as a still image i(n) (Step S429), displayed, and recorded.This extracts the image of the strongest lesion. Once the image of the strongest lesion is extracted, the variable n is changed to n=n+1 (step S430).

[0050] Figure 10 shows the image extraction process from the decline to the disappearance of the white reaction, and the calculation process for the time required until the appearance of the white reaction after acetic acid processing and the time required from the appearance to the disappearance of the white reaction. In step S431, one frame of the video is advanced, and in step S432, image analysis is performed on the frame image. Based on the image analysis, the CPU2 determines whether or not the disappearance of the white reaction has been detected (step S433). The disappearance of the white reaction may be determined by whether the white (yellow) area returns to the state of the acetic acid processed area in the image at the end of acetic acid processing, or by whether the color density and area of ​​the white (yellow) area fall below a predetermined level. If the CPU2 determines that the disappearance of the white reaction has not been detected (step S433: No), it then determines whether or not the decline of the white reaction is above a certain level (step S434). Here, whether or not the decline of the white reaction is above a certain level is determined by whether or not the amount of decline of the white (yellow) area relative to the image of the strongest lesion is above a predetermined value. Alternatively, the determination may be made based on whether the amount of decay of the white (yellow) area in the image one or several frames prior is greater than or equal to a predetermined value. If the amount of decay of the white reaction is not greater than or equal to a predetermined level (step S434: No), the process returns to step S431 and the processing from step S431 onwards is executed. If the amount of decay of the white reaction is greater than or equal to a predetermined level (step S434: Yes), the CPU2 determines that the white reaction is continuing while decaying, and outputs the frame image from the video as a still image i(n) (step S435), displays and records it, and changes the variable n to n=n+1 (step S436). When the variable n is changed, the process returns to step S431 and the processing from step S431 onwards is repeatedly executed. If it is determined in step S433 that the white reaction has subsided (step S433: Yes), the image of the white reaction subsiding is extracted as a still image i(n) (step S437), displayed, and recorded along with the playback time t2 of the video up to the still image i(n) (step S438). Subsequently, the time required t1-t0 from the acetic acid application to the white reaction to the appearance of the white reaction is calculated and recorded from the playback time t0 of the video up to the end of acetic acid application and the playback time t1 of the video up to the appearance of the white reaction (step S439).Furthermore, the time required from the appearance to the disappearance of the white reaction (t2-t1) is calculated from the video playback time (t2) up to the disappearance of the white reaction and recorded (step S440). With this, the image extraction process is completed.

[0051] In the second embodiment described above, a colposcopy camera was used as an example of the medical support device 1. However, the system is not limited to this, and as mentioned above, it may be a combination of an imaging device such as a colposcopy camera and a viewer device. Alternatively, it may be just a viewer device capable of capturing the captured images, or it may be an information processing device such as a personal computer.

[0052] In the above embodiment, the CPU 2 functioned as a control unit by executing the program 12 stored in the memory unit 4. However, instead of the CPU 2 executing the program 12 stored in the memory unit 4, dedicated hardware such as an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or various control circuits may be provided, and this dedicated hardware may function as a control unit. In this case, some parts may be implemented by the dedicated hardware, and other parts by software or firmware.

[0053] Furthermore, the above embodiment may be applied not only to the cervix, but also to other diseased areas such as the ear, nose, and throat or the digestive tract, or to any other drug solution other than acetic acid that is applied to make the condition of the lesion (such as color or shape) easier to observe.

[0054] Furthermore, in the above embodiment, the program 12 may be stored in the storage unit 4 in advance, or it may be stored by reading it from an external storage medium readable by a computer (such as a CD (Compact Disc)-ROM, DVD (Digital Versatile Disc)-ROM, memory card, USB memory, etc.) to the storage unit 4 via a storage medium reading unit. In addition, the program 12 can also be superimposed on a carrier wave and stored by reading it to the storage unit 4 via a communication medium such as the Internet.

[0055] This invention allows for various embodiments and modifications without departing from the broad spirit and scope of the invention. Furthermore, the above embodiments are for illustrative purposes only and do not limit the scope of the invention. In other words, the scope of the invention is indicated not by the embodiments, but by the claims. Various modifications made within the scope of the claims and the equivalent scope of the meaning of the invention are considered to be within the scope of this invention. The invention described in the original claims of this application is listed below. [Explanation of Symbols]

[0056] 1. Medical support device, 2. CPU, 3. RAM, 4. Memory unit, 5. Imaging unit, 6. Light-emitting unit, 7. Display unit, 8. Notification unit, 9. Operation unit, 10. Voice input unit, 11. Communication unit, 12. Program

Claims

1. A medical support device that assists in the performance of colposcopy, Based on multiple images obtained by continuous shooting of the target area to be subjected to acetic acid processing, the timing at which a predetermined change related to acetic acid processing occurs in the target area, or the image taken at that timing, The system includes a control unit that performs at least one of the following controls based on the specified timing or the specified image: control of shooting conditions in continuous shooting, control of image recording in continuous shooting, and control of display of the image recorded by continuous shooting. Medical support device.

2. The control unit, During the continuous shooting described above, an image is identified in which the predetermined change related to acetic acid processing occurs in the target area. As part of the aforementioned shooting condition control, during shooting, focus control is performed to focus on the target area where the predetermined change has occurred, based on the identified image. The medical support device according to claim 1.

3. The control unit, During the continuous imaging described above, the timing at which the predetermined change related to acetic acid processing occurs in the target area is identified. As part of the aforementioned shooting condition control, during shooting, lighting control is performed to switch the illumination light irradiated onto the target area at the specified timing. The medical support device according to claim 1.

4. The control unit, During the continuous imaging described above, the timing at which the predetermined change related to acetic acid processing occurs in the target area is identified. As the image recording control, a recording control is performed that selectively records images captured at the specified timing. The medical support device according to claim 1.

5. The control unit, After recording multiple images captured by the aforementioned continuous shooting, the image of the target area where the predetermined change related to acetic acid processing occurred is identified based on the multiple recorded images. The aforementioned display control involves performing a control to display the identified image. The medical support device according to claim 1.

6. A medical support device that assists in performing colposcopy, Based on multiple images obtained by continuous shooting of the target area to be subjected to acetic acid processing, the timing at which a predetermined change related to acetic acid processing occurs in the target area, or the image taken at that timing, Based on the specified timing or the specified image, control is performed on at least one of the following: control of shooting conditions in continuous shooting, control of image recording in continuous shooting, and control of display of images recorded by continuous shooting. Medical support methods.

7. A medical support device that assists in performing colposcopy, Based on multiple images obtained by continuous shooting of the target area to be subjected to acetic acid processing, the timing at which a predetermined change related to acetic acid processing occurs in the target area, or the image taken at that timing, Based on the specified timing or the specified image, control is performed on at least one of the following: control of shooting conditions in continuous shooting, control of image recording in continuous shooting, and control of display of images recorded by continuous shooting. A medical support program that enables the execution of a procedure.

Images