System and method for visualizing endoscopic image analysis result

Abstract

A method for visualizing an endoscopic image analysis result according to a first embodiment of the present invention comprises the steps in which: a model loading / condition setting unit loads a plurality of analysis models and sets analysis conditions; an image analysis unit analyzes an image by using the plurality of analysis models when image analysis is required; an activation map extracting / generating unit extracts activation maps (AM) of the plurality of analysis models and generates a common activation map (CAM) when the image analysis result satisfies the set analysis conditions, and generates an adjacent frame common activation map (ACAM) by applying a weight to the common activation map (CAM) extracted from the previous and current image frames; an activation value extracting unit extracts a maximum activation value from the adjacent frame common activation map (ACAM); and a closed curve extracting unit extracts a closed curve of the activation map when the maximum activation value is at least a threshold value, and draws the closed curve on the analysis image.

Description

Visualization system and method for endoscopic image analysis results

[0001] The present invention relates to a visualization system and method for endoscopic image analysis results, and more particularly, to a visualization system and method for endoscopic image analysis results that select endoscopic image analysis results using a plurality of artificial intelligence models and visualize the final results.

[0002] When analyzing endoscopic images using an AI (Artificial Intelligence) model to detect objects, a single-frame analysis can be used to determine if the target has been detected. However, using only single-frame results can reduce detection accuracy due to endoscope movement or false positives.

[0003] As an alternative to the above problems, a method has been proposed that determines detection of an analysis target when the analysis target is detected consecutively over a fixed number of frames, N. However, the processing speed of endoscopic image analysis devices can vary by manufacturer, and setting a fixed frame rate, N, can affect detection accuracy depending on the movement speed of the examiner operating the endoscope.

[0004] Endoscopy analysis software built on object detection models can output coordinates as results, representing them in a rectangular shape. To reduce false detections, postprocessing involves displaying the results on the screen when the analysis target is detected consecutively in N frames. Even when endoscopic analysis software is built with multiple object detection models, the rectangular coordinate output limits its ability to indicate the precise target area.

[0005] The present invention was created in consideration of the above matters, and the purpose of the present invention is to provide a visualization system and method for the results of endoscopic image analysis, which can firstly reduce the possibility of false detection predicted by each model by determining whether an analysis target is detected by using an agree all or voting method of the probability values ​​predicted by multiple models, and secondarily reduce the possibility of false detection by extracting a common activation map of the agree all or voting model for the current frame.

[0006] Another object of the present invention is to provide a visualization system and method for endoscopic image analysis results, which can reduce the possibility of false detection three-dimensionally by extracting an adjacent frame common activation area by applying a weight to the common activation area extracted from the previous frame and the current frame, and can reduce the possibility of false detection four-dimensionally by determining whether the peak activation value of the adjacent frame common activation area is greater than or equal to a detection confirmation threshold.

[0007] Another object of the present invention is to provide a visualization system and method for endoscopic image analysis results that can minimize false detections with multiple models, indicate the predicted analysis target in detail in the form of a closed curve, and select representative images from the analysis results by using common activation areas of adjacent frames in the representative image selection process.

[0008] In order to achieve the above purpose, the visualization system of the endoscopic image analysis results according to the present invention,

[0009] A model loading / condition setting unit that loads multiple analysis models and sets analysis conditions for the analysis models;

[0010] An image analysis unit that determines whether a situation requires image analysis, and if so, reads the image frame and analyzes the image using the plurality of analysis models;

[0011] An activation area extraction / generation unit that determines whether the image analysis result by the image analysis unit satisfies the analysis conditions set by the model loading / condition setting unit, and if the analysis conditions are satisfied, extracts the activation areas (activation maps, AM) of the plurality of analysis models, and creates a common activation area (common activation map, CAM) based on the extracted activation areas, and determines whether the analysis target has been detected in the previous image frame, and if detected, applies a weight to the common activation areas (CAM) extracted from the previous image frame and the current image frame to create an adjacent frame common activation area (ACAM);

[0012] An activation value extraction unit that extracts the maximum activation value from an adjacent frame common activation area (ACAM) generated by the above activation area extraction / generation unit;

[0013] A closed curve extraction unit that determines whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold, and if it is greater than or equal to the threshold, extracts a closed curve of the activation area and draws a closed curve on the corresponding analysis image; and

[0014] It is characterized in that it includes a control unit that checks the status and controls the operation of the model loading / condition setting unit, the image analysis unit, the activation area extraction / generation unit, the activation value extraction unit, and the closed curve extraction unit, sends control commands to cause the model loading / condition setting unit, the image analysis unit, the activation area extraction / generation unit, the activation value extraction unit, and the closed curve extraction unit to perform their respective functions, and reads data, information, and applications necessary for performing each function from a database (DB) and provides them, or stores them in the database (DB).

[0015] Here, when the above-described activation area extraction / generation unit generates the adjacent frame common activation area (ACAM), the adjacent frame common activation area (ACAM) can be generated using a method using a full agreement model or a method using a voting model.

[0016] At this time, the method using the above-mentioned full agreement model can set the analysis conditions of multiple analysis models to specific predicted probability values.

[0017] At this time, the method using the voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value that is different from the specific predicted probability value in the method using the overall agreement model.

[0018] In addition, it is preferable to further include a representative image selection unit that selects a representative image by measuring similarity with the adjacent frame common activation area (ACAM).

[0019] At this time, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value may be used as the probability of the representative image.

[0020] In addition, in order to achieve the above purpose, the visualization method of the endoscopic image analysis result according to the first embodiment of the present invention,

[0021] a) A step of loading multiple analysis models and setting analysis conditions of the analysis models by the model loading / condition setting unit;

[0022] b) A step in which the video analysis unit determines whether a situation requires video analysis, and if so, reads a video frame and analyzes the video using the plurality of analysis models;

[0023] c) A step of determining whether the image analysis result by the image analysis unit satisfies the analysis condition set by the model loading / condition setting unit by the activation area extraction / generation unit;

[0024] d) If the analysis condition is satisfied in the determination of step c), a step of the activation area extraction / generation unit extracting the activation area (activation map, AM) of the plurality of analysis models and generating a common activation area (common activation map, CAM) based on the extracted activation area;

[0025] e) a step of the above-described activation area extraction / generation unit determining whether an analysis target has been detected in a previous image frame, and if so, applying weights to common activation areas (CAMs) extracted from the previous image frame and the current image frame to generate an adjacent common activation area (ACAM);

[0026] f) a step of extracting the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit by the activation value extraction unit; and

[0027] g) The characteristic of the present invention is that it includes a step of determining whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold value, and if it is greater than or equal to the threshold value, extracting a closed curve of the activation area and drawing the closed curve on the corresponding analysis image.

[0028] Here, in the step e), when the activation area extraction / generation unit generates the adjacent frame common activation area (ACAM), the adjacent frame common activation area (ACAM) can be generated using a method using a full agreement model or a method using a voting model.

[0029] At this time, the method using the above-mentioned full agreement model can set the analysis conditions of multiple analysis models to specific predicted probability values.

[0030] At this time, the method using the voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value that is different from the specific predicted probability value in the method using the overall agreement model.

[0031] In addition, after the above step e), the representative image selection unit may further include a step of selecting a representative image by measuring similarity with the adjacent frame common activation area (ACAM).

[0032] At this time, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value may be used as the probability of the representative image.

[0033] In addition, in order to achieve the above purpose, a method for visualizing the endoscopic image analysis result according to the second embodiment of the present invention,

[0034] m) A step of loading multiple analysis models and setting analysis conditions of the analysis models by the model loading / condition setting unit;

[0035] n) A step in which the video analysis unit determines whether a situation requires video analysis, and if so, reads a video frame;

[0036] o) A step of determining whether the image analysis unit is in detection mode or classification mode, analyzing the image with a detection model if it is in detection mode, and analyzing the image with a classification model if it is in classification mode;

[0037] p) A step of determining whether the image analysis result by the image analysis unit satisfies the analysis condition set by the model loading / condition setting unit by the activation area extraction / generation unit;

[0038] q) If the analysis condition is satisfied in the determination of step d), a step of the activation area extraction / generation unit extracting the activation area (activation map, AM) of the plurality of analysis models and generating a common activation area (common activation map, CAM) based on the extracted activation area;

[0039] r) a step of the above-mentioned activation area extraction / generation unit determining whether an analysis target has been detected in a previous image frame, and if so, applying a weight to the common activation area (CAM) extracted from the previous image frame and the current image frame to generate an adjacent common activation area (ACAM);

[0040] s) A step in which the activation value extraction unit extracts the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit;

[0041] t) a step of the closed curve extraction unit determining whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold, and if it is greater than or equal to the threshold, extracting a closed curve of the activation area and drawing the closed curve on the corresponding analysis image; and

[0042] u) The feature is that the image analysis unit determines whether the analysis mode has been changed, and if the analysis mode has been changed, it includes a step of changing the current analysis mode to the detection mode if the current analysis mode has been changed, and to the classification mode if the current analysis mode has been changed, and to the detection mode if the current analysis mode has been changed.

[0043] Here, in the step r), when the activation area extraction / generation unit generates the adjacent frame common activation area (ACAM), the adjacent frame common activation area (ACAM) can be generated using a method using a full agreement model or a method using a voting model.

[0044] At this time, the method using the above-mentioned full agreement model can set the analysis conditions of multiple analysis models to specific predicted probability values.

[0045] At this time, the method using the voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value that is different from the specific predicted probability value in the method using the overall agreement model.

[0046] In addition, after the above step r), the representative image selection unit may further include a step of selecting a representative image by measuring similarity with the adjacent frame common activation area (ACAM).

[0047] At this time, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value may be used as the probability of the representative image.

[0048] According to the present invention, the detection of an analysis target is determined using a plurality of analysis models, a common activation area (CAM) of an overall agreement or voting model for the current frame is extracted, a weight is applied to the common activation area (CAM) extracted from the previous frame and the current frame to extract an adjacent frame common activation area (ACAM), and the possibility of false detection can be reduced four times by determining whether the maximum activation value of the adjacent frame common activation area (ACAM) is greater than or equal to a detection confirmation threshold, thereby having the advantage of reducing the possibility of false detection.

[0049] In addition, the analysis target predicted using multiple analysis models can be specified in detail in the form of a closed curve, and there is an advantage in that representative images can be selected from the analysis results by using the common activation area of ​​adjacent frames in the representative image selection process.

[0050] FIG. 1 is a drawing schematically showing the configuration of a visualization system for endoscopic image analysis results according to the present invention.

[0051] Figure 2 is a flowchart showing the execution process of a method for visualizing endoscopic image analysis results according to the first embodiment of the present invention.

[0052] FIG. 3 is a diagram illustrating an outline of a method for visualizing the results of an endoscopic image analysis according to the present invention, in which a common activation area of ​​adjacent frames is created by a full agreement method.

[0053] FIG. 4 is a diagram illustrating an outline of a method for visualizing an endoscopic image analysis result according to the present invention, in which a common activation area of ​​adjacent frames is created by a voting method.

[0054] Figure 5 is a flowchart showing the execution process of a method for visualizing endoscopic image analysis results according to a second embodiment of the present invention.

[0055] FIG. 6 is a diagram illustrating a representative image selection process through similarity measurement with common activation areas of adjacent frames in a visualization method of endoscopic image analysis results according to the present invention.

[0056] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0057] FIG. 1 is a schematic diagram illustrating the configuration of a visualization system for endoscopic image analysis results according to an embodiment of the present invention.

[0058] Referring to FIG. 1, a visualization system (100) of an endoscopic image analysis result according to the present invention may be configured to include a model loading / condition setting unit (110), an image analysis unit (120), an activation area extraction / generation unit (130), an activation value extraction unit (140), a closed curve extraction unit (150), and a control unit (160).

[0059] The model loading / condition setting unit (110) loads a plurality of analysis models (e.g., four) (see FIGS. 3 and 4) and sets analysis conditions of the analysis models (e.g., a predicted probability value of 0.8 or higher or three models having a predicted probability value of 0.85 or higher).

[0060] The video analysis unit (120) determines whether a situation requires video analysis, and if so, reads the video frame and analyzes the video using the plurality of analysis models.

[0061] The activation region extraction / generation unit (130) determines whether the image analysis result by the image analysis unit (120) satisfies the analysis conditions set by the model loading / condition setting unit (110), and if the analysis conditions are satisfied, extracts the activation regions (activation maps, AM) (see FIGS. 3 and 4) of the plurality of analysis models, generates a common activation region (CAM) based on the extracted activation regions (AM), determines whether the analysis target has been detected in the previous image frame, and if so, applies a weight to the common activation regions (CAM) extracted from the previous image frame and the current image frame to generate an adjacent frame common activation region (ACAM). Here, when the activation region extraction / generation unit (130) generates the adjacent frame common activation region (ACAM), the adjacent frame common activation region (ACAM) can be generated using an agree all model or a voting model. At this time, the method using the above-mentioned full agreement model can set the analysis conditions of multiple analysis models to a specific predicted probability value (for example, a predicted probability value of all multiple analysis models of 0.8 or higher). At this time, the method using the above-mentioned voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value different from the specific predicted probability value in the method using the above-mentioned full agreement model (for example, a predicted probability value of 0.85 or higher for three analysis models). Here, the specific predicted probability value as each analysis condition in the method using the above-mentioned full agreement model and the method using the voting model is not limited to being fixed to a single value, and can be set to various values ​​depending on the target or situation to which the analysis condition is applied.

[0062] The activation value extraction unit (140) extracts the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit (130).

[0063] The closed curve extraction unit (150) determines whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold, and if it is greater than or equal to the threshold, extracts a closed curve of the activation area and draws the closed curve on the corresponding analysis image.

[0064] The control unit (160) controls the status check and operation of the model loading / condition setting unit (110), the image analysis unit (120), the activation area extraction / generation unit (130), the activation value extraction unit (140), and the closed curve extraction unit (150), and transmits control commands to cause the model loading / condition setting unit (110), the image analysis unit (120), the activation area extraction / generation unit (130), the activation value extraction unit (140), and the closed curve extraction unit (150) to perform their respective functions, and reads data, information, and applications necessary for performing each function from a database (DB) (180) and provides them, or stores them in the database (DB) (180). Here, such a database (DB) (180) stores and manages various software programs for system operation, or data or information required when the model loading / condition setting unit (110), image analysis unit (120), activation area extraction / generation unit (130), activation value extraction unit (140) and closed curve extraction unit (150) perform functions or processes related to model loading and condition setting, image analysis, activation area extraction and generation, activation value extraction and closed curve extraction.

[0065] The visualization system (100) of the endoscope image analysis results according to the present invention having the above configuration may further include a representative image selection unit (170) that selects a representative image by measuring the similarity with the adjacent frame common activated area (ACAM). At this time, when the representative image selection unit (170) selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates greater than or equal to a threshold value may be used as the probability of the representative image. This will be described again later.

[0066] Here, the model loading / condition setting unit (110), image analysis unit (120), activation area extraction / generation unit (130), activation value extraction unit (140), closed curve extraction unit (150), control unit (160), representative image selection unit (170), and database (DB) (180) as described above may be integrated as a whole and configured as a single computer system.

[0067] Then, below, a method for visualizing endoscopic image analysis results based on a visualization system for endoscopic image analysis results according to the present invention having the above-described configuration will be described.

[0068] Figure 2 is a flowchart showing the execution process of a method for visualizing endoscopic image analysis results according to the first embodiment of the present invention.

[0069] Referring to FIG. 2, the method for visualizing the endoscopic image analysis results according to the first embodiment of the present invention first loads a plurality (e.g., four) analysis models (see FIGS. 3 and 4) through a model loading / condition setting unit (110) and sets analysis conditions of the analysis models (step S201). For example, the analysis conditions may be set such that the predicted probability value is 0.8 or higher, or three models have predicted probability values ​​of 0.85 or higher.

[0070] When the loading of the analysis model and the setting of the analysis conditions are completed in this way, the image analysis unit (120) determines whether a situation requires image analysis (step S202), and if a situation requires image analysis, reads the image frame and analyzes the image using the plurality of analysis models (steps S203 and S204).

[0071] Then, the activation area extraction / generation unit (130) determines whether the image analysis result by the image analysis unit (120) satisfies the analysis conditions set by the model loading / condition setting unit (110) (step S205).

[0072] If the analysis conditions are satisfied in the determination of the above step S205, the activation area extraction / generation unit (130) extracts the activation areas (activation maps, AM) of the multiple analysis models (see FIGS. 3 and 4) (step S206) and generates a common activation area (common activation map, CAM) based on the extracted activation areas (AM) (step S207).

[0073] In addition, the above-described activation region extraction / generation unit (130) determines whether an analysis target (e.g., a specific lesion such as gastric cancer) has been detected in the previous image frame (frame N-1) (see FIGS. 3 and 4) (step S208), and if so, applies a weight to the common activation region (CAM) extracted from the previous image frame and the current image frame to generate an adjacent frame common activation region (ACAM) (step S209). Here, when the above-described activation region extraction / generation unit (130) generates the adjacent frame common activation region (ACAM), the adjacent frame common activation region (ACAM) can be generated using a method using a global agreement model (see FIG. 3) or a method using a voting model (see FIG. 4). At this time, the method using the global agreement model can set the analysis conditions of multiple analysis models to a specific predicted probability value (e.g., all of the multiple analysis models have a predicted probability value of 0.8 or higher). At this time, the method using the voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value (e.g., a predicted probability value of 0.85 or higher of three analysis models) that is different from the specific predicted probability value in the method using the overall agreement model. Here, the specific predicted probability value as each analysis condition in the method using the overall agreement model and the method using the voting model as described above is not limited to being fixed to a single value as described above, and can be set to various values ​​depending on the target or situation to which the analysis condition is applied. Here, the generation of the adjacent frame common activation region (ACAM) using the overall agreement model and the voting model as described above will be described again later.

[0074] When the adjacent frame common activation area (ACAM) is generated in this way, the activation value extraction unit (140) extracts the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit (130) (step S210).

[0075] Thereafter, the closed curve extraction unit (150) determines whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold value (step S211), and if it is greater than or equal to the threshold value, extracts a closed curve of the activation area and draws the closed curve on the corresponding analysis image (steps S212 and S213). By doing so, the endoscopic image analysis results are visualized, and through this, the analysis target predicted using multiple analysis models can be indicated in detail in the form of a closed curve.

[0076] Meanwhile, in the above series of processes, after step S209, a step may be further included in which the representative image selection unit (170) selects a representative image by measuring similarity with the adjacent frame common activation area (ACAM).

[0077] At this time, when the representative image selection unit (170) selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates exceeding a threshold value may be used as the probability of the representative image. This will be explained later.

[0078] FIG. 3 is a diagram illustrating an outline of a method for visualizing the results of an endoscopic image analysis according to the present invention, in which a common activation area of ​​adjacent frames is created by a full agreement method.

[0079] Referring to FIG. 3, the activation area extraction / generation unit (130) first preprocesses the endoscopic images of the previous image frame (Frame N-1) and the current image frame (Frame N), and then extracts the activation areas (AM) of multiple analysis models (e.g., four of Model 1 to Model 4). At this time, the analysis conditions of each analysis model can be set to a predicted probability value of 0.8 or higher. FIG. 3 exemplifies that in the case of the previous image frame (Frame N-1), Model 1 has a predicted probability value of 0.83, Model 2 has a predicted probability value of 0.89, Model 3 has a predicted probability value of 0.92, and Model 4 has a predicted probability value of 0.87, respectively, and in the case of the current image frame (Frame N), Model 1 has a predicted probability value of 0.86, Model 2 has a predicted probability value of 0.88, Model 3 has a predicted probability value of 0.91, and Model 4 has a predicted probability value of 0.87, respectively.

[0080] When the activation area (AM) is extracted in this way, the activation area extraction / generation unit (130) calculates the average value of the predicted probability values ​​of each activation area (AM) of multiple (4) activation areas of the previous image frame (Frame N-1) and the current image frame (Frame N), and generates a common activation area (CAM) based on each activation area (AM) of the previous image frame (Frame N-1) and the current image frame (Frame N) having the average value.

[0081] After that, a weight is applied to each common activation region (CAM) extracted from the previous image frame (Frame N-1) and the current image frame (Frame N) to generate an adjacent frame common activation region (ACAM).

[0082] FIG. 4 is a diagram illustrating an outline of a method for visualizing an endoscopic image analysis result according to the present invention, in which a common activation area of ​​adjacent frames is created by a voting method.

[0083] Referring to Fig. 4, this process is essentially identical to the process of creating a common activation area of ​​adjacent frames using the full agreement method described with reference to Fig. 3 above. Therefore, the description of the process of creating a common activation area of ​​adjacent frames will be replaced with the description in Fig. 3 above, and only the differences from the full agreement method described above will be described.

[0084] This voting method sets the analysis conditions for each analysis model differently from the analysis conditions of the overall agreement method described above. In other words, this voting method (method using a voting model) differs in that it sets the analysis conditions for multiple analysis models to a specific predicted probability value (e.g., a predicted probability value of 0.85 or higher for three analysis models) that is different from the specific predicted probability value (predicted probability value of 0.8 or higher) of the overall agreement method (method using a overall agreement model) described above. In Fig. 4, for the previous video frame (Frame N-1), Model 1 shows a predicted probability value of 0.83, Model 2 shows a predicted probability value of 0.89, Model 3 shows a predicted probability value of 0.92, and Model 4 shows a predicted probability value of 0.87, respectively. However, according to the analysis condition (predicted probability values ​​of the three analysis models are 0.85 or higher), the activation area (AM) of Model 1 with a predicted probability value of 0.83 is discarded, and only the activation areas (AM) of the remaining three analysis models (Models 2 to 4) with predicted probability values ​​of 0.85 or higher are taken.

[0085] Figure 5 is a flowchart showing the execution process of a method for visualizing endoscopic image analysis results according to a second embodiment of the present invention.

[0086] Referring to FIG. 5, the method for visualizing the results of endoscopic image analysis according to the second embodiment of the present invention first has a model loading / condition setting unit (110) load a plurality of analysis models (e.g., four) (see FIGS. 3 and 4) and set analysis conditions of the analysis models (step S501). For example, the analysis conditions can be set such that the predicted probability value is 0.8 or higher, or three models have predicted probability values ​​of 0.85 or higher.

[0087] In this way, when the loading of the analysis model and the setting of the analysis conditions are completed, the image analysis unit (120) determines whether a situation requires image analysis (step S502), and if a situation requires image analysis, reads the image frame (step S503).

[0088] And the image analysis unit (120) determines whether it is a detection mode or a classification mode (step S504), and if it is a detection mode, it analyzes the image using a detection model (step S505), and if it is a classification mode, it analyzes the image using a classification model (step S506).

[0089] Then, the activation area extraction / generation unit (130) determines whether the image analysis result by the image analysis unit (120) satisfies the analysis conditions set by the model loading / condition setting unit (110) (step S507).

[0090] If the analysis conditions are satisfied in the determination of the above step S507, the activation area extraction / generation unit (130) extracts the activation areas (AM) of the multiple analysis models (see FIGS. 3 and 4) (step S508) and generates a common activation area (CAM) based on the extracted activation areas (AM) (step S509).

[0091] In addition, the above-described activation region extraction / generation unit (130) determines whether an analysis target (e.g., a specific lesion such as gastric cancer) has been detected in the previous image frame (frame N-1) (see FIGS. 3 and 4) (step S510), and if so, applies a weight to the common activation region (CAM) extracted from the previous image frame and the current image frame to generate an adjacent frame common activation region (ACAM) (step S511). Here, when the above-described activation region extraction / generation unit (130) generates the adjacent frame common activation region (ACAM), the adjacent frame common activation region (ACAM) can be generated using a method using a global agreement model (see FIG. 3) or a method using a voting model (see FIG. 4). At this time, the method using the global agreement model can set the analysis conditions of multiple analysis models to a specific predicted probability value (e.g., all of the multiple analysis models have a predicted probability value of 0.8 or higher). At this time, the method using the voting model can also set the analysis conditions of multiple analysis models to a specific predicted probability value (e.g., a predicted probability value of 0.85 or higher of three analysis models) that is different from the specific predicted probability value in the method using the overall agreement model. Here, the specific predicted probability value as each analysis condition in the method using the overall agreement model and the method using the voting model is not limited to being fixed to a single value as described above, and can be set to various values ​​depending on the target or situation to which the analysis conditions are applied.

[0092] When the adjacent frame common activation area (ACAM) is generated in this way, the activation value extraction unit (140) extracts the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit (130) (step S512).

[0093] The closed curve extraction unit (150) determines whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold (step S513), and if greater than or equal to the threshold, extracts a closed curve of the activation area and draws the closed curve on the corresponding analysis image (steps S514, S515).

[0094] Thereafter, the image analysis unit (120) determines whether the analysis mode has been changed (step S516), and if the analysis mode has been changed, if the current analysis mode is detection mode, it changes to classification mode, and if the current analysis mode is classification mode, it changes to detection mode (step S517).

[0095] In a series of processes as described above, after step S511, a step may further be included in which the representative image selection unit (170) selects a representative image by measuring similarity with the adjacent frame common activation area (ACAM).

[0096] At this time, when the representative image selection unit (170) selects a representative image, the output probability of the frame with the closest similarity may be used as the probability of the representative image, or the average output probability of representative image candidates greater than or equal to a threshold value may be used as the probability of the representative image.

[0097] Meanwhile, FIG. 6 is a drawing showing a representative image selection process through similarity measurement with common activation areas of adjacent frames in a visualization method of endoscopic image analysis results according to the present invention.

[0098] Referring to FIG. 6, if there are 9 ACAM frames starting from the M-th frame (frame M) to the (M+1-th frame (frame M+1), ..., M+8-th frame (frame M+8) as similar ACAM frames for the common activation area (ACAM) frame of the last adjacent frame as the current frame (frame N) or the ACAM frame with the maximum probability, it is assumed that the ACAM similarity of each frame is 0.89, 0.93, 0.86, 0.88, 0.87, 0.87, 0.84, 0.79, 0.78, respectively, and the probabilities are 0.92, 0.94, 0.87, 0.90, 0.91, 0.90, 0.89, 0.86, 0.87, respectively.

[0099] At this time, when the representative image selection unit (170) selects a representative image, the output probability (0.94) of the frame (frame M+1) with the closest similarity (similarity 0.93) among the representative image candidates can be used as the probability of the representative image, or the average output probability value (0.90) of the representative image candidates that are greater than the threshold value can be used as the probability of the representative image.

[0100] As described above, the visualization system and method for the endoscopic image analysis results according to the present invention has the advantage of reducing the possibility of false detection four times by determining whether an analysis target is detected using a plurality of analysis models, extracting a common activation area (CAM) of an overall agreement or voting model for the current frame, applying a weight to the common activation area (CAM) extracted from the previous frame and the current frame to extract an adjacent frame common activation area (ACAM), and determining whether the maximum activation value of the adjacent frame common activation area (ACAM) is greater than or equal to a detection confirmation threshold.

[0101] In addition, the analysis target predicted using multiple analysis models can be specified in detail in the form of a closed curve, and there is an advantage in that representative images can be selected from the analysis results by using the common activation area of ​​adjacent frames in the representative image selection process.

Claims

1. A model loading / condition setting section that loads multiple analysis models and sets analysis conditions of the analysis models; A video analysis unit that determines whether a situation requires video analysis, and if so, reads a video frame and analyzes the video using the plurality of analysis models; An activation area extraction / generation unit that determines whether the image analysis result by the image analysis unit satisfies the analysis conditions set by the model loading / condition setting unit, and if the analysis conditions are satisfied, extracts the activation areas (activation maps, AM) of the plurality of analysis models, and creates a common activation area (common activation map, CAM) based on the extracted activation areas, and determines whether the analysis target has been detected in the previous image frame, and if detected, applies a weight to the common activation areas (CAM) extracted from the previous image frame and the current image frame to create an adjacent frame common activation area (ACAM); An activation value extraction unit that extracts the maximum activation value from an adjacent frame common activation area (ACAM) generated by the above activation area extraction / generation unit; A closed curve extraction unit that determines whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold, and if it is greater than or equal to the threshold, extracts a closed curve of the activation area and draws a closed curve on the corresponding analysis image; and A visualization system for endoscopic image analysis results, comprising a control unit that checks the status and controls the operation of the model loading / condition setting unit, the image analysis unit, the activation area extraction / generation unit, the activation value extraction unit, and the closed curve extraction unit, and transmits control commands to cause the model loading / condition setting unit, the image analysis unit, the activation area extraction / generation unit, the activation value extraction unit, and the closed curve extraction unit to perform their respective functions, and reads data, information, and applications necessary for performing each function from a database (DB) and provides them, or stores them in the database (DB).

2. In paragraph 1, A visualization system for endoscopic image analysis results, characterized in that the above-mentioned activation area extraction / generation unit generates the adjacent frame common activation area (ACAM) using a method using a full agreement model or a method using a voting model.

3. In paragraph 2, A visualization system for endoscopic image analysis results, characterized in that the method using the above-mentioned full agreement model sets the analysis conditions of multiple analysis models to specific predicted probability values.

4. In paragraph 3, A visualization system for endoscopic image analysis results, characterized in that the method using the above voting model sets the analysis conditions of multiple analysis models to a specific predicted probability value different from the specific predicted probability value in the method using the above overall agreement model.

5. In paragraph 1, A visualization system for endoscopic image analysis results, further comprising a representative image selection unit for selecting a representative image through a similarity measurement with the adjacent frame common activation area (ACAM).

6. In paragraph 5, A visualization system for endoscopic image analysis results, characterized in that, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity is used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value is used as the probability of the representative image. 7.a) A step of loading multiple analysis models and setting analysis conditions of the analysis models by the model loading / condition setting section; b) A step in which the video analysis unit determines whether a situation requires video analysis, and if so, reads a video frame and analyzes the video using the plurality of analysis models; c) A step of determining whether the image analysis result by the image analysis unit satisfies the analysis condition set by the model loading / condition setting unit by the activation area extraction / generation unit; d) If the analysis condition is satisfied in the determination of step c), a step of the activation area extraction / generation unit extracting the activation area (activation map, AM) of the plurality of analysis models and generating a common activation area (common activation map, CAM) based on the extracted activation area; e) a step of the above-described activation area extraction / generation unit determining whether an analysis target has been detected in a previous image frame, and if so, applying weights to common activation areas (CAMs) extracted from the previous image frame and the current image frame to generate an adjacent common activation area (ACAM); f) a step of extracting the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit by the activation value extraction unit; and g) A method for visualizing the results of an endoscopic image analysis, comprising a step of: a closed curve extraction unit determining whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold; and, if greater than or equal to the threshold, extracting a closed curve of an activated area and drawing the closed curve on the corresponding analysis image.

8. In paragraph 7, A method for visualizing the results of endoscopic image analysis, characterized in that in the step e), the activation area extraction / generation unit generates the adjacent frame common activation area (ACAM) using a method using a full agreement model or a method using a voting model.

9. In paragraph 8, A method for visualizing endoscopic image analysis results using the above-mentioned full agreement model, characterized in that the analysis conditions of multiple analysis models are set to specific predicted probability values.

10. In paragraph 9, A method for visualizing endoscopic image analysis results, characterized in that the method using the above voting model sets the analysis conditions of multiple analysis models to a specific predicted probability value different from the specific predicted probability value in the method using the above overall agreement model.

11. In paragraph 7, A method for visualizing the results of endoscopic image analysis, further comprising a step of selecting a representative image by a representative image selection unit after the above step e) by measuring similarity with the adjacent frame common activated area (ACAM).

12. In paragraph 11, A method for visualizing the results of endoscopic image analysis, characterized in that, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity is used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value is used as the probability of the representative image. 13.m) A step of loading multiple analysis models and setting analysis conditions of the analysis models by the model loading / condition setting section; n) A step in which the video analysis unit determines whether a situation requires video analysis, and if so, reads a video frame; o) A step of determining whether the image analysis unit is in detection mode or classification mode, analyzing the image with a detection model if it is in detection mode, and analyzing the image with a classification model if it is in classification mode; p) A step of determining whether the image analysis result by the image analysis unit satisfies the analysis condition set by the model loading / condition setting unit by the activation area extraction / generation unit; q) If the analysis condition is satisfied in the determination of step d), a step of the activation area extraction / generation unit extracting the activation area (activation map, AM) of the plurality of analysis models and generating a common activation area (common activation map, CAM) based on the extracted activation area; r) a step of the above-mentioned activation area extraction / generation unit determining whether an analysis target has been detected in a previous image frame, and if so, applying a weight to the common activation area (CAM) extracted from the previous image frame and the current image frame to generate an adjacent common activation area (ACAM); s) A step in which the activation value extraction unit extracts the maximum activation value from the adjacent frame common activation area (ACAM) generated by the activation area extraction / generation unit; t) a step of the closed curve extraction unit determining whether the extracted maximum activation value is greater than or equal to a detection confirmation threshold, and if it is greater than or equal to the threshold, extracting a closed curve of the activation area and drawing the closed curve on the corresponding analysis image; and u) A method for visualizing the results of endoscopic image analysis, including a step of determining whether the image analysis unit has changed the analysis mode, and if the analysis mode has changed, changing the current analysis mode to the classification mode if the current analysis mode is the detection mode, or to the detection mode if the current analysis mode is the classification mode.

14. In paragraph 13, A method for visualizing the results of endoscopic image analysis, characterized in that in the step r), the activation area extraction / generation unit generates the adjacent frame common activation area (ACAM) using a method using a full agreement model or a method using a voting model.

15. In paragraph 14, A method for visualizing endoscopic image analysis results using the above-mentioned full agreement model, characterized in that the analysis conditions of multiple analysis models are set to specific predicted probability values.

16. In paragraph 15, A method for visualizing endoscopic image analysis results, characterized in that the method using the above voting model sets the analysis conditions of multiple analysis models to a specific predicted probability value different from the specific predicted probability value in the method using the above overall agreement model.

17. In paragraph 13, A method for visualizing the results of endoscopic image analysis, further comprising a step of selecting a representative image by a representative image selection unit after the above step r) by measuring similarity with the adjacent frame common activated area (ACAM).

18. In paragraph 17, A method for visualizing the results of endoscopic image analysis, characterized in that, when the representative image selection unit selects a representative image, the output probability of the frame with the closest similarity is used as the probability of the representative image, or the average output probability of representative image candidates greater than a threshold value is used as the probability of the representative image.

Images