Visualization result display control device, learning process visualization system and learning process visualization method

Abstract

A visualization result display control device comprises a display control unit (55) for performing display control of matching and displaying a first similar section overlay display (V521) and a second similar section overlay display (V522), wherein the first similar section overlay display (V521) shows a first display (D661) overlaid on a misclassification sample (D211), the first display (D661) visualizes a first similar section of the misclassification sample on an extraction sample (D251) that is a sample with a feature similar to the misclassification sample, wherein the misclassification sample (D211) is a sample that has been misclassified into a prediction class by an object capture model, and the second similar section overlay display (V522) shows a second display (D662) overlaid on the The extraction sample being displayed shows,and the second display (D662) visualizes a second similar section of the extraction sample on the misclassification sample.

Description

TECHNICAL AREA

[0001] The present disclosure relates to a learning process visualization technique for visualizing and displaying a machine learning learning process. BACKGROUND ON THE STATE OF THE TECHNOLOGY

[0002] In recent years, machine learning techniques have been used to detect objects in the real world. However, from the perspective of the machine learning mechanism, the process leading to the prediction is a black box, and it is difficult to implement measures to improve the accuracy of the machine learning. An explainable AI (XAI) technique is proposed to address this problem. For example, non-patent literature 1 proposes a method for visualizing a prediction base using gradient information calculated from a feature set of a last convolutional layer of a convolutional neural network (CNN) and a predicted evaluation of the CNN in an object detection model. REFERENCE LIST NON-PATENT LITERATURE

[0003] Non-Patent Literature 1: Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, “Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization”. ar-Xiv:1610.02391 (2016) SUMMARY OF THE INVENTIONAL PROBLEM

[0004] However, existing techniques require complex processing to display prediction bases, and therefore the problem is that if only one prediction base is displayed for an incorrect prediction result, only part of the displayed content can be understood, and there remains room for improvement in the object capture model.

[0005] The present disclosure was made to solve such a problem, and one purpose of the present disclosure is to provide a learning process visualization technique which can help to make suggestions for improving an object acquisition model. SOLUTION TO THE PROBLEM

[0006] An aspect of a visualization result display control device according to an embodiment of the present disclosure comprises a display control unit for performing display control of matching and displaying a first-similar section overlay display and a second-similar section overlay display, wherein the first-similar section overlay display shows a first display overlaid on a misclassification sample, the first display visualizes a first similar section of the misclassification sample on an extraction sample, which is a sample with a feature similar to the misclassification sample, wherein the misclassification sample is a sample that has been misclassified into a prediction class by an object acquisition model, and the second-similar section overlay display shows a second display overlaid on the extraction sample.and the second display visualizes a second similar section of the extraction sample on the misclassification sample. ADVANTAGEOUS EFFECTS OF THE INVENTION

[0007] A visualization result display control device according to an embodiment of the present disclosure can help to provide suggestions for improving an object acquisition model. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a hardware configuration representation showing a configuration of a learning process visualization system. Fig. Figure 2 is a functional block diagram showing a functional configuration of a pattern extraction device. Fig. Figure 3 is a functional block diagram showing a functional configuration of a visualization result display control device. Fig. Figure 4 is a functional block representation showing a functional configuration of a similar-section specification device. Fig. Figure 5 is a functional block representation showing a functional configuration of a misclassification BBox extraction device. Fig. Figure 6A is a diagram showing a configuration example of the hardware of each facility that forms the learning process visualization system. Fig. Figure 6B is a diagram showing a configuration example of the hardware of each facility that forms the learning process visualization system. Fig. Figure 7 is a representation that shows an example of a display formed from visualization information. Fig. Figure 8 is a representation illustrating an example of a misclassification pattern display. Fig. Figure 9 is a representation that illustrates an example of a display format for a misclassification BBox group. Fig. Figure 10 is a representation that illustrates an example of a display formed from visualization information. Fig. Figure 11 is a representation that shows an example of a partially captured sample. Fig. Figure 12 is a flowchart that illustrates how the learning process visualization system works. DESCRIPTION OF THE EXECUTION FORMS

[0008] Embodiments of the present disclosure are described in detail below with reference to the accompanying drawings. It should be noted that components designated in the drawings with identical or similar reference numerals have identical or similar components or functions, and that redundant descriptions of such components are omitted.

[0009] Furthermore, the term “or” in this disclosure means an inclusive “or” unless specifically stated otherwise. Any instance in which the term “or” is used to mean an exclusive “or” will be expressly stated. Design 1.<Erläuterung des gesamten Systems>

[0010] A learning process visualization system 1 according to embodiment 1 of the present disclosure is described with reference to Fig. 1 explained Fig. Figure 1 is a hardware configuration representation that depicts a configuration of the learning process visualization system 1 according to the present embodiment. As shown in Fig. As shown in Figure 1, the learning process visualization system 1 comprises a sample extraction device 2, an operation input device 3, a storage device 4, a similar section specification device 6, a misclassification BBox group extraction device 7, a visualization result display control device 5, and a display device 8. The sample extraction device 2 is a device that extracts a sample with a feature similar to a BBox (referred to in some cases below as the "misclassification sample") D211, which has been misclassified by an object acquisition model that acquires an object. As previously known, the object acquisition model is a type of machine learning model. The operation input device 3 is a device that accepts input from a user of the learning process visualization system 1 to operate the system.Storage facility 4 is a facility that stores existing data and the machine learning model. Similar section specification facility 6 is a facility that specifies a similar section between the misclassification sample D211 and a selection sample D651. Note that the selection sample refers to a sample chosen from one, two, or more extraction samples extracted by sample extraction facility 2, based on user input accepted through operation input facility 3. Misclassification BBox group extraction facility 7 is a facility that extracts a group of BBox groups, including all misclassification BBoxes. Note that BBox is an abbreviation for bounding box, bounding frame, or bounding field.The visualization result display control device 5 is a device that controls the display of a visualization result on the display device 8. The display device 8 is a device that displays the result according to the display control of the visualization result display control device 5. The entire control between these devices can be performed by a control device not shown, which is included in the learning process visualization system 1, or by a special device, such as the visualization result display control device 5, between the devices shown. Fig. The facilities shown in section 1 will be used. <Erläuterung jeder Komponente>

[0011] The following section describes in more detail the components that make up the learning process visualization system 1. [Sample extraction unit 2]

[0012] Sample extraction device 2 is described with reference to Fig. 2 explained in more detail. Fig. Figure 2 is a functional block diagram showing a functional configuration of the sample extraction device 2. As shown in Fig. As shown in Figure 2, the sample extraction device 2 comprises a misclassification sample acquisition unit 21, a feature set acquisition unit 22, an existing image feature acquisition unit 23, a feature similarity calculation unit 24, a sample extraction unit 25 and an extraction sample output unit 26. (Misclassification Sample Acquisition Unit 21)

[0013] The misclassification sample acquisition unit 21 reads the misclassification sample D211, which is a misclassification BBox D741 selected by the user from a misclassification BBox group D751, as well as an original image D212 containing the misclassification sample D211 and training data D213.

[0014] The misclassification BBox group is a group of misclassification samples containing one, two, or more misclassification samples. A misclassification sample is a BBox that specifies the position and size of an object that the object detection model has determined belongs to a class that differs from a correct class in an image containing a multitude of objects. It is sufficient that an area inside the BBox contains at least part of a target object; it does not need to contain the entire object. The misclassification sample (BBox) is included in the misclassification BBox group along with an original image containing the BBox and associated training data.In one example, such a misclassification BBox group is created in advance by the misclassification BBox group extraction facility 7 and stored in advance in storage facility 4. In another example, the misclassification BBox group can be stored in an unillustrated storage facility located outside the learning process visualization system 1.

[0015] The misclassification sample acquisition unit 21 outputs the read original image D212, the misclassification sample D211, and the training data D213 to the extraction sample output unit 26. Additionally, the misclassification sample acquisition unit 21 outputs the misclassification sample D211 to the feature set acquisition unit 22. At this point, the misclassification sample acquisition unit 21 can output the original image D212 to the feature set acquisition unit 22 instead of the misclassification sample D211. (Feature set - Acquisition unit 22)

[0016] The feature set acquisition unit 22 extracts a misclassification sample feature set D221, which is a feature set of a misclassification sample, from the misclassification sample D211. For example, a trained machine learning model such as a convolutional neural network (CNN) can be used to extract the misclassification sample feature set D221. (Existing image feature acquisition unit 23)

[0017] The existing image feature acquisition unit 23 reads all existing image feature sets D232, D233, ..., and D23n, which were pre-stored, from an existing image database F231, which is a database for existing images. The existing image database F231 is pre-stored, for example, in storage device 4. The existing image database F231 contains an existing image file F2311, which is a file of an existing image; a feature set file F2312 of the existing image, which is a file of a feature set of the existing image; and training data F2313 of the existing image. It should be noted that the existing image data consists, for example, of image data used for training or fitting a model. The existing image feature acquisition unit 23 outputs the read existing image feature sets D232, D233, ..., and D23n to the feature similarity calculation unit 24. (Feature similarity calculation unit 24)

[0018] The feature similarity calculation unit 24 calculates the similarity between the misclassification sample feature set D221, which is input by the feature set acquisition unit 22, and each of the existing image feature sets D232, D233, ..., and D23n, which are input by the existing image feature acquisition unit 23, and outputs similarities D241, D242, ..., and D24n between the misclassification sample and the existing image feature sets to the sample extraction unit 25. Here, similarity D241 denotes the similarity between the misclassification sample feature set D221 and the existing image feature set D232, similarity D242 denotes the similarity between the misclassification sample feature set D221 and the existing image feature set D233, and similarity D24n denotes the similarity between the misclassification sample feature set D221 and the existing image feature set D23n.To calculate similarity, a general method for calculating similarity, such as the Euclidean distance or cosine similarity, can be used.

[0019] Furthermore, the feature similarity calculation unit 24 also outputs the existing image file F2312 to the sample extraction unit 25. (Sample extraction unit 25)

[0020] The sample extraction unit 25 extracts one or more types of extraction samples D251, D252, ... and D25n based on all similarities D241, D242, ... and D24n input by the feature similarity calculation unit 24, and outputs the extracted extraction samples D251, D252, ... and D25n to the extraction sample output unit 26.

[0021] To extract the extraction samples D251, D252, ..., and D25n, one or more search patterns R251, R252, ..., and R25n are predefined. The sample extraction unit 25 searches for a multitude of existing images in an order of higher similarity to an existing image (existing image file F2311) and outputs these found existing images as the extraction samples D251, D252, ..., and D25n to the extraction sample output unit 26. It should be noted that the sample extraction unit 25 acquires the existing images from the existing image feature acquisition unit 23 or the feature similarity calculation unit 24. Such processing is performed on all search patterns. It should be noted that the number of samples to be extracted per search pattern can be freely specified by the user.

[0022] The extraction sample can be a sample that has a feature similar to a feature of a misclassification sample, or a sample that has a feature similar to a feature of the training data of the misclassification sample.

[0023] The sample extraction unit 25 can search for an extraction sample from the existing image file F2311 or from a misclassification BBox group that is not the misclassification sample D211. (Extraction sample output unit 26)

[0024] The extraction sample output unit 26 can output all or part of the misclassification sample D211, the original image D212, and the training data D213 inputted by the misclassification sample acquisition unit 21, as well as the extraction samples D251, D252, ..., and D25n inputted by the sample extraction unit 25, outside of the sample extraction unit 2, based or not on a user selection input. The user selection input is provided via the operation input unit 3, and the sample extraction unit 2 accepts this selection input. It should be noted that one or more samples selected by the user from the extraction samples D251, D252, ..., and D25n are referred to as selection samples D651, D652, ..., and D65n. [Operation input device 3]

[0025] The operational input device 3 is a device that accepts user input and is, for example, a keyboard or a mouse. The operational input device 3 sends the accepted user input to another device, such as the sample extraction device 2. [Storage device 4]

[0026] The storage device 4 stores various pieces of information and is implemented as a storage device such as a hard disk. In the present embodiment, the storage device 4 stores a trained model used to derive a test image or to extract a feature, and stores a test image database F41 and the existing image database F231. The test image database F41 stores a test image file F411 and training data F412 of an existing image. The test image is, for example, an image used for testing, such as an image used to evaluate the accuracy of a trained model. [Visualization result display control unit 5]

[0027] Fig. Figure 3 is a configuration diagram illustrating a functional configuration of the visualization result display control device 5 according to the present embodiment. As shown in Fig. 3 the visualization result display control unit 5 includes a misclassification sample reading unit 51, a selection sample reading unit 53, a similar section image reading unit 54, a display content education unit 52 and a display control unit 55. (Misclassification Sample - Reading Unit 51)

[0028] The misclassification sample reading unit 51 reads the misclassification sample D211, which is output by the sample extraction unit 2, and sends the read misclassification sample D211 to the display content formation unit 52. (Selection sample - Reading Unit 53)

[0029] The selection sample reading unit 53 reads the selection sample D651, which was output by the sample extraction unit 2, and sends the read selection sample D651 to the display content formation unit 52. (Similar Section Image Reading Unit 54)

[0030] The similar section reading unit 54 reads a misclassification sample similar section image D6611, which was input by the similar section specification unit 6, and one or more selection sample similar section images D6621, D6631, ... and D66nN, and sends the read misclassification sample similar section image D6611 and one or more selection sample similar section images D6621, D6631, ... and D66nN to the display content formation unit 52. (Display content-Educational Unit 52)

[0031] The display content formation unit 52 forms content output by the misclassification sample reading unit 51, the selection sample reading unit 53 and the similarity section image reading unit 54 in a user-friendly format and outputs the formed content to the display control unit 55. (Display control unit 55)

[0032] The display control unit 55 controls the display of the generated content on the display device 8. The display device 8 is, for example, a liquid crystal display device.

[0033] The visualization result display control unit 5 forms each visualization information input from the sample extraction unit 2 and the similar section specification unit 6 and displays the formed visualization information D511. Details of the formed visualization information D511 are described later. [Similar-Section-Specification Facility 6]

[0034] Fig. Figure 4 is a functional block diagram showing a functional configuration of the similar-section specification device 6 according to the present embodiment. As shown in Fig. As shown in Figure 4, the similar section specification unit 6 comprises a misclassification image acquisition unit 61, a feature set acquisition unit 62, a feature similarity calculation unit 63, a similar section specification unit 64, a selection sample acquisition unit 65, a similar section image generation unit 66, and a similar section image output unit 67. (Misclassified image acquisition unit 61)

[0035] The misclassification image acquisition unit 61 reads the user-selected misclassification sample D211 from the misclassification BBox group D751 and sends the read misclassification sample D211 to the feature set acquisition unit 62. (Feature set - Acquisition unit 62)

[0036] The feature acquisition unit 62 acquires an image feature set D621 of a misclassification sample from the misclassification sample D211, which was read by the misclassification image acquisition unit 61; acquires the image feature sets D622, D623, ... and D62n of one or more selection samples from one or more selection samples D651, D652, ... and D65n, which were input by the selection sample acquisition unit 65; and outputs the acquired image feature sets (D621, D622, D623, ... and D62n) to the feature similarity calculation unit 63. For example, a trained machine learning model such as a convolutional neural network (CNN) can be used to acquire the feature set. (Feature similarity calculation unit 63)

[0037] The feature similarity calculation unit 63 calculates similarities D631, D632, ..., and D63n between the feature set D621 of the misclassification sample and the feature sets D622, D623, ..., and D62n of the selection samples for each channel, and outputs the calculated similarities D631, D632, ..., and D63n to the similarity section specification unit 64. A general method for calculating similarity, such as Euclidean distance or cosine similarity, can be used. It should be noted that a channel refers to a variable that stores information about a color, texture, and the like, and represents a feature space of a pixel, and comprises three channels: one channel representing R (red), one channel representing G (green), and one channel representing B (blue), for example, in the case of an RGB color space. (Similar Section Specification Unit 64)

[0038] The Similar Section Specification Unit 64 outputs a channel corresponding to the greatest similarity among the similarities from the information of similarities D631, D632, ... and D63n, which was input by the Feature Similarity Calculation Unit 63, as Similar Section Explanation Channels D641, D642, ... and D64n to the Similar Section Image Generation Unit 66. It should be noted that, in the case of an RGB image, similarity D631, for example, includes similarity D631R of the R channel, similarity D631G of the G channel, and similarity D631B of the B channel. The same applies to other D632, ... and D63n. (Selection sample acquisition unit 65)

[0039] The selection sample acquisition unit 65 reads one or more samples (the selection sample D651, D652, ..., and D65n) that have been selected by the user from the extraction samples and outputs the read selection samples to the feature set acquisition unit 62. (Similar Section Generation Unit 66)

[0040] Based on the information from one or more similar section explanation channels D641, D642, ... and D64n, which are input by the similar section specification unit 64, the similar section image generation unit 66 creates a heatmap D661 (first display) showing a similar section (first similar section) of the misclassification sample, and heatmaps D662, D663, ... and D66n (second displays) showing similar sections (second similar sections) of one or more selection samples, and integrates the misclassification sample D211 and the one or more selection samples D651, D652, ... and D65n...., with the respective heatmaps. An image obtained after the integration of each sample is referred to as the misclassification sample similar section image D6611 and as the selection sample similar section images D6621, D6631, ... and D66nN.This means that the misclassification sample similar section image D6611 is an image obtained by integrating the misclassification pattern D211 and the heatmap D661, and the selection sample similar section images D6621, D6631, ..., and D66nN are images obtained by integrating the selection samples D651, D652, ..., and D65n and the heatmaps D662, D663, ..., and D66n. The similar section image generation unit 66 outputs the misclassification sample similar section image D6611 and the selection sample similar section images D6621, D6631, ..., and D66nN to the similar section image output unit 67. The heatmaps can be heatmaps to which transparency is assigned depending on the similarity differences. (Similar Section Image Output Unit 67)

[0041] The Similar Section Image Output Unit 67 outputs the misclassification sample similar section image D6611, which was accepted by the similar section image generation unit 66, and the selection sample similar section images D6621, D6631, ..., and D66nN to another facility. [Misclassification BBox Group Extraction Unit 7]

[0042] Fig. Figure 5 is a functional block diagram showing a functional configuration of the misclassification B-box extraction device 7 according to the present embodiment. As shown in Fig. As shown in Figure 5, the misclassification BBox group extraction unit 7 comprises a test image acquisition unit 71, an inference unit 72, a training data reading unit 73, a true / false determination unit 74, and a misclassification BBox group output unit 75. (Test pattern acquisition unit 71)

[0043] The test pattern acquisition unit 71 reads test pattern data D711 from the test pattern database F41 stored in the storage device 4 and outputs the read test pattern data D711 to the inference unit 72. (Inference Unit 72)

[0044] The inference unit 72 inputs the test image data D711, received from the test image acquisition unit 71, into an inference engine and outputs an inference result D721, which is an inference result of the inference engine, to the true / false determination unit 74. For example, a trained machine learning model such as a convolutional neural network (CNN) can be used as the inference engine. The inference result D721 includes the test image data D711, a BBox that specifies the position and size of a detected object, and a prediction class obtained by predicting the class membership of the detected object. (Training data reading unit 73)

[0045] The training data reading unit 73 reads training data D731, that is, information about a position, a label and the like of the object, which are assigned to each test image, from the test image database F41, which is stored in the memory device 4, and outputs the read training data D731 to the true / false determination unit 74. (True / False determination unit 74)

[0046] The True / False Determination Unit 74 performs a True / False determination on the prediction class of each BBox based on the inference result D721 input by the inference unit 72 and the training data D731 input by the training data read unit, and outputs the misclassification BBox D741 and the test image data D711, which is an original image including the misclassification BBox D741, to the misclassification BBox group output unit 75, using a BBox that has been determined to be false as the misclassification BBox D741. (Misclassification BBox Group Output Unit 75)

[0047] The misclassification BBox group output unit 75 integrates, as the misclassification BBox group D751, the one or more misclassification BBoxes D741 input by the true / false determination unit 74, the test image data D711 (which is the original image including the misclassification BBoxes D741), and the training data D731, and outputs the integrated misclassification BBox group D751 to another device of the learning process visualization system 1. In this case, other data, such as the reliability associated with the misclassification BBoxes D741, can also be included in the misclassification BBox group D751. <Detaillierte Erläuterung von Merkmalsabschnitt> [Educated visualization information]

[0048] Next, the generated visualization information D511 of the visualization result display control unit 5 is displayed with reference to the Fig. 7 and Fig. 10 described. The Fig. 7 and Fig. Figure 10 shows an example of a display of the generated visualization information D511. (Formed visualization information D511)

[0049] In the present embodiment, the visualization information D511 comprises a display V51 showing the misclassification sample D211, one or more types of displays V53, V54, V55, ... and V5n showing one or more types of extraction samples D251, D252, ... and D25n similar to the misclassification sample D211, and a display V52 showing a similar section between the misclassification sample D211 and the selection sample D651, which was selected from the extraction samples D252, ... and D25n. The display control unit 55 can control the display of pieces of information contained in the visualization information D511 across a variety of areas, thus enabling the identification of the pieces of information contained in the visualization information D511. As described in Fig. As shown in Figure 7, the display control unit 55 can, for example, synchronize and display the displays V51, V53, V54, V55, ... and V5n in different ranges. Although the range in which display V51 is shown in the example is in Fig. While the V51 display is located at the top level (level 7), it can also be displayed in a different area, such as at the bottom level or in a break.

[0050] The V52 display can be part of the V51 display and part of one or more types of the V53, V54, V55, ..., and V5n displays. As in Fig. As shown in 7, display V52 can, for example, be a display that is part of display V51X, which is part of display V51 in Fig. 7 is, and a V53X display, which is part of the V53 display in Fig. 7 is, includes. The display V51 and one or more types of displays V53, V54, V55, ..., and V5n are aligned and displayed in such a way that the display V51X and the display V53X, which form the display V52, are also aligned and displayed.

[0051] Furthermore, display V52 can be a display formed by extracting and forming part of display V51 and part of one or more types of displays V53, V54, V55, ... and V5n, and may differ from display V51 or from one or more types of displays V53, V54, V55, ... and V5n. For example, as in Fig. 10 shows that display V52 is a display that is created by comparing side-by-side with display V51X, which is part of display V51. Fig. 7 is, and the V53X display, which is part of the V53 display in Fig. 7 is formed. In the following, each display that forms the visualization information D511 is described in more detail in a new paragraph.

[0052] It is sufficient that the V51X display and the V53X display contained within the V52 display are aligned and displayed, either side by side or separately. Furthermore, the V51X and V53X displays can be aligned horizontally, vertically, or diagonally. (Misclassification test indicator V51)

[0053] Fig. Figure 8 is a representation which shows a detailed configuration example for the misclassification probe display V51 in Fig. 7 illustrates. As in Fig. As shown in Figure 8, the misclassification sample display V51 includes a misclassification BBox group display V511, which displays a misclassification BBox group, a misclassification result display D512, which displays a misclassification result, a misclassification sample scale display V513, which displays a misclassification sample, and a BBox information display V514, which displays information about a BBox. The display control unit 55 can perform display control of the display of information from the misclassification BBox group display V511, the misclassification result display D512, the misclassification sample scale display V513 and the BBox information display V514 in different areas to enable the identification of the misclassification BBox group display V511, the misclassification sample scale display D512, the misclassification sample scale display V513 and the BBox information display V514. [Misclassification BBox group display V511)

[0054] The Misclassification BBox Group Display V511 is a list-format display of the misclassification BBox groups obtained from the aforementioned Misclassification BBox Group Extraction Unit 7. An example of the display format is as follows. However, the display format must not be this specific format. ■List display format

[0055] As in Fig. As shown in Figure 9, a parent element V5111 displays a correct class, and a child element V5112 belonging to the parent element V5111 displays a group of misclassified BBoxes. ■Element name of child element V5112“ <vorhersageklasse>_<Korrekte Klasse>_<Zuverlässigkeit>_ <originalbild-dateiname>"

[0056] As explained above, the misclassification BBox group display V511 is a list-format display obtained by grouping a set of misclassification sample groups, including a multitude of misclassification samples per correct class. The display control unit 55 controls the display of this misclassification BBox group display V511 on the display device 8. (Misclassification result display D512)

[0057] The misclassification result display D512 is a display obtained by integrating a test image (original image 212) including the misclassification BBox (misclassification sample) D211, which was selected by the user, the training data D213, and a misclassification BBox (misclassification sample D211). As in Fig. Figure 8 shows the original image D212 integrally, which displays the training data D213 and the position and size of the misclassification B-Box (misclassification sample D211). The display control unit 55 controls the display of the misclassification result indicator D512. (Misclassification Sample Scale Display V513)

[0058] The misclassification sample scale display V513 is a scale display that contains a view obtained by zooming in or out on the misclassification sample D211. In a case where an image showing a similar section is superimposed on the misclassification sample V211, the misclassification sample scale display V513 is a scale display that contains a view where the similar section superimposed view is enlarged or reduced. The zooming in or out can be performed based on a user operation. The display control unit 55 performs display control of the misclassification sample scale display V513. (BBox information display V514)

[0059] The BBox information display V514 shows related information pertaining to the misclassification sample. This display includes a prediction class (V5141), a correct class (V5142), and a misclassification type (V5143) for the sample. The misclassification type (V5143) informs the user about the type of misclassification. For example, the misclassification type (V5143) displays information such as "Misclassification of class," "Misclassification due to partial object capture," "Misclassification due to capture of a completely different object," or similar. It should be noted that information other than that listed above can also be added to the misclassification samples. (Similar section display V52)

[0060] Fig. Figure 10 is a diagram illustrating a configuration example for the Similar Section Display V52. In the example in Fig. 10. The similar section display D6611, in which the heatmap D661, which shows a similar section between the misclassification sample D211 and the selection sample D651, was overlaid on the misclassification sample D211, is displayed as a similar section overlay display V521 (First Similar Section Overlay Display), and the similar section display D6621, in which the heatmap D662, which shows a similar section between the selection sample D651 and the misclassification sample D211, was overlaid on the selection sample D651, is displayed as a selection sample similar section overlay display V522 (Second Similar Section Overlay Display).As described above, the Similar Section Display V52 comprises the Similar Section Display Display V521, which is a display of a similar section image obtained by displaying an image showing a similar section of a misclassification sample on the misclassification sample, and one or more Selection Sample Similar Section Display Displays V522, V523, ... and V52n, which are displays obtained by displaying an image showing a similar section of a selection sample on the selection sample. (Selection sample displays V53, V54, V55, ..., and V5n)

[0061] The extraction sample displays V53, V54, V55, ... and V5n are displays of samples that exhibit a higher similarity to the misclassification sample D211. A sample exhibiting a higher similarity to the misclassification sample D211 is sought by the sample extraction unit 2 using one or more search patterns R251. That is, the sample extraction unit 2 extracts samples with a higher similarity to the misclassification sample D211 in an order of one sample with a higher similarity from the existing image database F231 or the misclassification BBox group using the one or more search patterns R251, and the visualization result display control unit 5 displays the extraction samples D251, D252, ... and D25n, which are the samples extracted by the sample extraction unit 2, as the extraction sample displays V53, V54, V55, ... and V5n.It should be noted that any number of extraction samples will be displayed. Below are three examples of extraction samples according to each search pattern: one where V53 represents a "similar sample", one where V54 represents a "partially captured sample", and one where V55 represents a "correct sample". (Misclassification test: V53)

[0062] Data whose prediction class is the same as that of misclassification sample D211 and whose color, texture, or shape is similar to misclassification sample D211 are displayed as similar samples (V53). For example, in a case where the prediction class of misclassification sample D211 is "dog," an image with the class "dog" is extracted from the existing image database F231 and displayed in order of a higher similarity to that of misclassification sample D211. (Partially captured sample: V54)

[0063] As in Fig. As shown in Figure 11, a misclassification BBox in which both a predicted class and a correct class match those of misclassification sample D211, and in which an object is partially captured, is displayed as a partially captured sample (V54). For example, if the predicted class of misclassification sample D211 is "Dog" and the correct class is "Person", a misclassification BBox containing both the predicted class "Dog" and the correct class "Person," and in which an object is partially captured, is extracted from the misclassification BBox group in an order of a misclassification BBox group with a higher similarity to that of misclassification sample D211 and displayed. (Correct sample: V55)

[0064] Data whose correction class matches that of the misclassification sample D211 and whose color, texture, or shape resembles that of the misclassification sample D211 are displayed as a correct sample (V55). For example, in a case where the correct class of the misclassification sample D211 is "Person", an image with the class "Person" is extracted from existing images in an order of one image that has a higher similarity to that of the misclassification sample D211 and displayed.

[0065] Next, an example of the hardware configuration of the facilities included in the learning process visualization system 1 will be presented, with reference to the Fig. 6A and Fig. 6B described. Each function of the sample extraction device 2, the visualization result display control device 5, the similar section specification device 6, and the misclassification BBox group extraction device 7 among the devices included in the learning process visualization system 1 is implemented by a processing circuit. The processing circuit may be a Fig. 6A depicts a dedicated processing circuit 100a, or a processor 100b, which is located in a Fig. The 6B displayed memory executes 100c stored programs.

[0066] In a case where the processing circuit is the dedicated processing circuit 100a, the dedicated processing circuit 100a corresponds, for example, to a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a combination thereof. The functions of the above devices, which are included in the learning process visualization system 1, can be implemented by a variety of different processing circuits, or the functions of the devices can be implemented collectively by a single processing circuit.

[0067] In a case where the processing circuit is the processor 100b, the functions of the above-mentioned devices included in the learning process visualization system 1 are implemented by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the working memory 100c. The processor 100b implements the function of each device by reading and executing the programs stored in the working memory 100c. Examples of working memory 100c include non-volatile or volatile semiconductor memory such as random-access memory (RAM), read-only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), or electrically erasable programmable read-only memory (EEPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini-disk, and a DVD. <Beschreibung der Funktionsweise>[Learning Process Visualization System 1]

[0068] The functionality of the learning process visualization system 1 is explained. A process from the input of a misclassification BBox group into the learning process visualization system 1 to the visualization of a similar section by the user is described with reference to... Fig. 12 described. Fig. Figure 12 is a flowchart illustrating a series of visualization processes of the learning process visualization system 1. (Step ST1)

[0069] In one example, such a misclassification BBox group 751 is created in advance by the misclassification BBox group extraction device 7 and stored in advance in the storage device 4. Under this assumption, the sample extraction device 2 of the learning process visualization system 1 reads the misclassification BBox group 751 from the storage device 4 in step ST1. The sample extraction device 2 performs a display control of the output of the read misclassification BBox group 751 to the visualization result display control device 5, converts the accepted misclassification BBox group 751 into a list format or the like by the visualization result display control device 5, and displays the converted misclassification BBox group 751 on the display device 8. (Step ST2)

[0070] In step ST2, the user selects a specific misclassification BBox from misclassification BBox group 751 via the operation input device 3. This BBox is displayed on the display device 8. This selected misclassification BBox is referred to as misclassification sample D211. The misclassification sample acquisition unit 21 of the sample extraction device 2 acquires this misclassification sample D211 and outputs the acquired misclassification sample D211 to the feature set acquisition unit 22. At this point, the misclassification sample acquisition unit 21 can output the original image D212 to the feature set acquisition unit 22 instead of the misclassification sample D211.

[0071] Furthermore, the misclassification sample acquisition unit 21 also acquires the original image D212 and the training data D213 of the misclassification sample D211. In addition, the sample extraction unit 2 sends the original image D212, the misclassification sample D211, and the training data D213 to the visualization result display control unit 5, and the visualization result display control unit 5 performs a display control of the display of the original image D212, the misclassification sample D211, and the training data D213 on the display unit 8. (Step ST3)

[0072] In step ST3, the feature set acquisition unit 22 reads the misclassification sample feature set D221. At this point, the feature set acquisition unit 22 can read a feature set from the original image D212 instead of the misclassification sample D211. For example, a trained machine learning model such as a convolutional neural network (CNN) can be used to read the feature set. Additionally, the existing image feature acquisition unit 23 reads one or more types of existing image features D232, D233, ..., and D23n, which are stored in memory device 4. (Step ST4)

[0073] In step ST4, the feature similarity calculation unit 24 calculates the similarities between the misclassification sample feature set D221 and one or more types of existing image feature sets D232, D233, ..., and D23n. (Step ST5)

[0074] In step ST5, the sample extraction unit 25 determines whether the search is complete based on all predefined search patterns R251, R252, and E25n. If the search is complete based on all search patterns R251, R252, and E25n (step ST5; Yes), processing continues with step ST8. Conversely, if the search is not complete based on all search patterns R251, R252, and E25n (step ST5; No), processing continues with step ST6. (Step ST6)

[0075] In step ST6, the sample extraction unit 25 searches for the existing image file F2311, which satisfies one or more search patterns R251, R252, and E25n, within an order of the existing image file F2311 with the highest similarity to the misclassification sample D211. The extracted samples, as described above, are designated as the extraction samples D251, D252, ..., and D25n. At this point, the sample extraction unit 25 can search for samples within misclassification BBox groups and extract those that differ from the misclassification sample D211, instead of the existing image file F2311, within an order of a sample with a higher similarity to the misclassification sample D211. (Step ST7)

[0076] In step ST7, the sample extraction unit 25 outputs any number of the one or more extraction samples D251, D252, ..., and D25n extracted in step ST6 to the visualization result display control unit 5 in an order of an extraction sample with a higher similarity. (Step ST8)

[0077] In step ST8, the learning process visualization system 1 determines whether the user has selected an exit button via the operation input device 3. If the exit button has been selected (step ST8; Yes), the learning process visualization system 1 terminates the processing. Conversely, if the exit button has not been selected (step ST8; No), the processing continues with step ST9. (Step ST9)

[0078] In step ST9, the learning process visualization system 1 determines whether the user has selected one or more extraction samples D251, D252, ..., and D25n via the operation input device 3. If extraction sample D251 has been selected (step ST9; Yes), processing continues with step ST10. Conversely, if extraction sample D251 has not been selected (step ST9; No), processing continues with step ST11. (Step ST10)

[0079] Step ST10 comprises one step (referred to as "Step ST10-1") performed by the Similar Section Specifier 6 and one step (referred to as "Step ST10-2") performed by the Visualization Result Display Control Unit 5. In Step ST10-1, the Similar Section Specifier 6 specifies a similar section between the misclassification sample D211 and the selected extraction sample D251, converts the misclassification sample D211 into an image (Similar Section Image D6611) with the heatmap D661 showing the similar section, converts the selected extraction sample D251 into an image (Similar Section Image D6621) with the heatmap D662 showing the similar section, and outputs the converted images to the Visualization Result Display Control Unit 5.

[0080] In other words, in step ST10-1, the Similar Section Specification Unit 6 generates the Similar Section Display Display V521, which displays a display (D661) that visualizes the similar section of the misclassification sample D211 on the extraction sample D251, which is a sample with a feature similar to the misclassification sample D211, on the misclassification sample D211, and the Similar Section Display Display V522, which displays a display (D662) that visualizes a similar section of the extraction sample D251 on the misclassification sample D211, on the extraction sample D251.

[0081] In step ST10-2, the visualization result display control unit 5 performs a display control of matching and displaying the converted images on the display unit 8. That is, in step ST10-2, the visualization result display control unit 5 performs a display control of matching and displaying the generated similar section overlay display V521 and the generated similar section overlay display V522. (Step ST11)

[0082] In step ST11, the learning process visualization system 1 determines whether the user has selected another misclassification sample from the misclassification BBox group 751 via the operation input device 3. If the additional misclassification sample has been selected (step ST11; Yes), processing continues with step ST2. Conversely, if the additional misclassification sample has not been selected (step ST11; No), processing continues with step ST8. <Erläuterung der Wirkung>

[0083] According to the above functionality, the learning process visualization system 1, according to the present embodiment, extracts samples from the existing image database or the misclassification BBox group that have features similar to the misclassification samples, i.e., samples that were misclassified by the object acquisition model. The learning process visualization system 1 compares and displays the samples extracted as misclassification samples (extraction samples) and overlays a display that visualizes similar sections between the misclassification samples and the extraction samples on both the misclassification samples and the extraction samples.The user who sees such a display can easily identify which feature influenced the misclassification, so that the learning process visualization system 1 can help to provide the user with suggestions for improving the object detection model.

[0084] In particular, the similar section between the misclassification sample and the extraction sample is indicated by displaying a visual representation that visualizes the similar section without using a dedicated index or term. This makes it possible to achieve an effect that allows even users without specialized knowledge to examine the improvement of the object detection model. Accordingly, the learning process visualization system 1, as disclosed herein, is suitable, for example, for deployment at a site where AI for object detection is to be implemented, but no AI engineer is available.In a case where, for example, the learning process visualization system 1, which is a system that captures birds, is introduced, if the birds are misclassified as different objects, the learning process visualization system 1 presents the user with image features such as color, texture, shape, and the like that cause the misclassification by using existing images or images from another misclassification BBox. Consequently, the user can understand which image feature led to a misclassification, even without specialized knowledge.

[0085] It should be noted that the embodiments can be combined, and that each embodiment can be modified or omitted as required. INDUSTRIAL APPLICABILITY

[0086] A learning process visualization technique according to the present disclosure can be used as a technique for improving an object acquisition model. Reference symbol list

[0087] 1: Learning Process Visualization System, 2: Sample Extraction Unit, 3: Operation Input Unit, 4: Memory Unit, 5: Visualization Result Display Control Unit, 6: Similar Section Specification Unit, 7: Misclassification BBox Group Extraction Unit, 8: Display Unit, 21: Misclassification Sample Acquisition Unit, 22: Feature Set Acquisition Unit, 23: Existing Image Feature Acquisition Unit, 24: Feature Similarity Calculation Unit, 25: Sample Extraction Unit, 26: Extraction Sample Output Unit, 51: Misclassification Sample Read Unit, 52: Display Contents Formation Unit, 53: Selection Sample Read Unit, 54: Similar Section Image Read Unit, 55: Display Control Unit, 61: Misclassification Image Acquisition Unit, 62: Feature set acquisition unit, 63: Feature similarity calculation unit, 64: Similar section output unit, 65: Selection sample acquisition unit,66: Similar section image generation unit, 67: Similar section image output unit, 71: Test image acquisition unit, 72: Inference unit, 73: Training data read unit, 74: True / False determination unit, 75: Misclassification BBox output unit, 100a: Processing circuit, 100b: Processor, 100c: RAM, QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited non-patent literature

[0000] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, Dhruv Batra, „Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization". ar-Xiv:1610.02391 (2016

[0003] < / vorhersageklasse>

Claims

[1] Visualization result display control device, comprising a display control unit for performing display control of matching and displaying a first similar section overlay display and a second similar section overlay display, wherein the first-similar section display shows a first display that is displayed on a misclassification sample, the first display visualizes a first similar section of the misclassification sample on an extraction sample that is a sample with a feature similar to the misclassification sample, wherein the misclassification sample is a sample that has been misclassified into a prediction class by an object capture model, and The Second Similar Section Display shows a second display that is displayed on the extraction sample, and the second display visualizes a second similar section of the extraction sample on the misclassification sample. [2] Learning process visualization system, comprehensive: the visualization result display control device according to claim 1; and a similar section specification device to specify the first similar section and generate the first similar section overlay display, and to specify the second similar section and generate the second similar section overlay display. [3] Learning process visualization system according to claim 2, wherein The similar section specification device calculates the first similar section and the second similar section using a feature set from the misclassification sample and a feature set from the extraction sample, and A heatmap with transparency is displayed as the first display and the second display to generate the First Similar Section overlay display and the Second Similar Section overlay display. [4] Learning process visualization system according to claim 3, wherein the similarity section specification device calculates a plurality of similarities of a plurality of channels between the feature set of the misclassification sample and the feature set of the extraction sample, and creates the heatmap using a similarity of a channel with a highest similarity among the plurality of calculated similarities. [5] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a sample with a feature similar to a feature of the misclassification sample from an existing image database, which is a database of existing images, and extracts a found sample as the extraction sample. [6] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a sample having a feature similar to a feature of the misclassification sample from a misclassification boundary field group, including a group of misclassified boundary fields, and extracts a found sample as the extraction sample. [7] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a sample with a feature similar to a feature of training data of the misclassification sample from an existing image database, which is a database of existing images, and extracts a found sample as the extraction sample. [8] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a sample having a feature similar to a feature of training data of the misclassification sample from a misclassification bounding field group including a group of misclassification bounding fields, and extracts a found sample as the extraction sample. [9] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a similar sample belonging to the prediction class of the misclassification sample and extracts a similar sample found as the extraction sample. [10] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a partially captured sample and extracts a found partially captured sample as the extraction sample, wherein the partially captured sample is a sample belonging to the same correct class and prediction class as a correct class and prediction class of the misclassification sample, and is obtained by partially capturing an object. [11] Learning process visualization system according to one of claims 2 to 4, further comprising a sample extraction device for searching for and extracting the extraction sample, wherein the sample extraction device searches for a correct sample and extracts a found correct sample as the extraction sample, wherein the correct sample belongs to the same class as a correct class of the misclassification sample. [12] Learning process visualization system according to one of claims 5 to 11, wherein the display control unit further performs a display control of grouping a group of misclassification sample groups containing a plurality of misclassification samples by correct class in order to display them as a list. [13] Learning process visualization system according to one of claims 5 to 12, wherein the display control unit further performs display control of the integration and display of an original image and of training data of a misclassification sample which is a target of the first similar section overlay display, and a misclassification sample which is a target of the first similar section overlay display. [14] Learning process visualization system according to one of claims 5 to 13, wherein the display control unit further performs display control of the magnification and display of the first similar section overlay display. [15] Learning process visualization system according to one of claims 5 to 14, wherein the display control unit further performs display control of the display of associated information belonging to a misclassification sample which is a target of the first similar section display. [16] Learning process visualization system according to claim 15, wherein the associated information includes information about the prediction class. [17] Learning process visualization system according to claim 15, wherein the associated information contains information about a correct class. [18] Learning process visualization system according to claim 15, wherein the associated information contains information about a type of misclassification. [19] Learning process visualization method of a learning process visualization system comprising a similar section specification device and a visualization result display control device, wherein the learning process visualization method comprises: Generate, through the Similar Section Specification Device, a First Similar Section Display Indicator and a Second Similar Section Display Indicator, wherein the first-similar section display shows a first display that is displayed on a misclassification sample, the first display visualizes a first similar section of the misclassification sample on an extraction sample that is a sample with a feature similar to the misclassification sample, wherein the misclassification sample is a sample that has been misclassified into a prediction class by an object capture model, and The second similar section display shows a second display that is overlaid on the extraction sample; the second display visualizes a second similar section of the extraction sample on the misclassification sample; and Perform, through the visualization result display control device, a display control of matching and displaying the generated first similar section overlay display and the generated second similar section overlay display.

Images