Program, information processing apparatus, and information processing method

Abstract

To allow more easy check of a result of estimation made by a machine learning model.SOLUTION: A program causes a computer to execute a classification step, a re-classification step, a determination step, and an output control step. The classification step executes classification processing of inputting first input data that is input data for estimation to a first model to obtain a classification class for each of a plurality of first components included in the first input data. The re-classification step executes re-classification processing of obtaining, for each of the plurality of first components, a re-classification class into which the first component is classified, from among a plurality of re-classification classes, by using the classification class and additional information. The determination step determines, for at least part of the plurality of re-classification classes, an output mode of the first components classified into the re-classification class. The output control step controls output of the first components according to the determined output mode.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a program, an information processing device, and an information processing method. [Background technology]

[0002] As a technology for confirming the inference results (classification results) of a machine learning model that performs a classification task, a visualization technology has been proposed in which a different color is assigned to each inference result and each component included in the input data is displayed using the color assigned to the inference result for that component. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-044674 [Patent Document 2] Japanese Patent Application Publication No. 2020-103880 [Patent Document 3] International Publication No. 2023 / 032480 [Non-patent literature]

[0004] [Non-Patent Document 1] Vijay Badrinarayanan et al., “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation”, arxiv: 1511.00561, 2016. Summary of the Invention [Problem to be solved by the invention]

[0005] The present invention aims to provide a program, an information processing device, and an information processing method that enable estimation results obtained by a machine learning model to be more easily confirmed. [Means for solving the problem]

[0006] According to an embodiment, the present invention provides a program for causing a computer to execute a classification step, a reclassification step, a determination step, and an output control step. The classification step executes a classification process in which first input data, which is input data for estimation, is input to a first model, and a classification class is obtained for each of a plurality of first components included in the first input data. The reclassification step executes a reclassification process in which, for each of the plurality of first components, a reclassification class is obtained from the plurality of reclassification classes into which the first component is classified, using the classification class and additional information. The determination step determines an output mode of the first component classified into the reclassification class for at least some of the plurality of reclassification classes. The output control step controls the output of the first component according to the determined output mode. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram of an information processing apparatus according to a first embodiment. [Figure 2] 10 is a flowchart of a visualization process according to the first embodiment. [Figure 3] FIG. 10 is a diagram for explaining an example of visualization processing. [Figure 4] FIG. 10 is a block diagram of an information processing apparatus according to a second embodiment. [Figure 5] 10 is a flowchart of a visualization process according to the second embodiment. [Figure 6] FIG. 10 is a block diagram of an information processing apparatus according to a third embodiment. [Figure 7] 10 is a flowchart of a visualization process according to the third embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a display screen that displays data. [Figure 9] FIG. 10 is a diagram showing an example of a display screen that displays data. [Figure 10] FIG. 10 is a diagram showing an example of a display screen that displays data. [Figure 11] FIG. 10 is a diagram showing an example of a display screen that displays data. [Figure 12] FIG. 10 is a diagram showing an example of a display screen that displays data. [Figure 13] FIG. 1 is a hardware configuration diagram of an information processing apparatus according to first to third embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0008] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of an information processing apparatus according to the present invention will be described in detail below with reference to the accompanying drawings.

[0009] With the visualization technology described above, color coding is performed based only on the classified estimation results, making it impossible to check additional information other than the estimation results. For example, in order to check whether the estimation results are correct (an example of additional information), it is necessary to visualize the true values ​​using the same method as the estimation results, and to make it possible to visually compare the two visualized results of the estimation results and the true values. With this method, the ease of enumeration and comparison can be reduced by preparing additional information to be compared for each visualized data. Furthermore, because the comparison is done visually, there is a high possibility that something will be overlooked.

[0010] Therefore, in the following embodiment, the estimation results (classification results) from the classification task are reclassified using additional information, and the output mode (color, etc.) of each component of the input data is determined based on the reclassification results. This makes it possible to check both the estimation results and the additional information using only a single visualization result. In other words, it becomes easier to check the estimation results from a machine learning model.

[0011] In the following, we will mainly explain an example where the input data is 3D point cloud data and the classification task is semantic segmentation.

[0012] Three-dimensional point cloud data is data that includes information indicating the three-dimensional position of each point in a three-dimensional space. In three-dimensional point cloud data, points correspond to components, and values ​​indicating the position of each point correspond to element values. Three-dimensional point cloud data may also include element values ​​for each point other than position (such as color information).

[0013] The input data is not limited to three-dimensional point cloud data, and may be any data that includes element values ​​for multiple component elements. Other examples of input data are described below. 3D mesh data: The mesh (mesh-shaped area) corresponds to the component, and the value indicating the mesh position corresponds to the element value. Image data: A pixel corresponds to a component, and a pixel value corresponds to an element value. Image data may be two-dimensional image data or three-dimensional image data. Image data may also be still image data or moving image data (video). In the case of moving image data, it can also be interpreted that an image at each time (frame) corresponds to a component, and the pixel value of that image corresponds to an element value. Audio data: The audio at each time corresponds to a component, and the value of the audio (waveform data, etc.) corresponds to an element value.

[0014] The three-dimensional point cloud data, three-dimensional mesh data, and three-dimensional image data correspond to three-dimensional input data including element values ​​for each of a plurality of components corresponding to three-dimensional positions. The point cloud data and mesh data may be two-dimensional (two-dimensional point cloud data, two-dimensional mesh data).

[0015] The input data may be acquired by any method. For example, if the input data is three-dimensional point cloud data, the three-dimensional point cloud data may be acquired by a sensor such as a three-dimensional laser scanner.

[0016] Semantic segmentation is the task of determining the class of each point, which is a component of 3D point cloud data, that indicates the type of object to which that point belongs.

[0017] The classification task is not limited to semantic segmentation, and may be any other task that can provide an estimation result for each component. Other examples of classification tasks are described below. -Classification tasks for video data, such as classifying scenes and characters in each frame - Classification tasks that classify scenes and speakers at each time using audio data

[0018] (First embodiment) In the first embodiment, true values ​​of the estimation results are used as additional information. For example, the information processing device of this embodiment reclassifies the estimation results obtained by semantic segmentation of 3D point cloud data using separately obtained true values, and visualizes the reclassification results.

[0019] 1 is a block diagram showing an example of the configuration of an information processing device 100 according to the first embodiment. As shown in FIG. 1, the information processing device 100 includes a storage unit 120, a display unit 131, a classification unit 101, a reclassification unit 102, a determination unit 103, and an output control unit 104.

[0020] The storage unit 120 stores various types of information used in the information processing device 100. For example, the storage unit 120 stores input data 121 and true value data 122. The input data 121 is data that serves as input to a machine learning model that executes a classification task, and is, for example, three-dimensional point cloud data.

[0021] The true value data 122 is data that represents the true value of the estimation result of the classification task. The true value data 122 may be obtained by any method. For example, a method of manually setting the true value or a method of estimating the true value (pseudo true value) by an estimation method (such as a method using a different model) other than the machine learning model used in the classification task can be applied.

[0022] The storage unit 120 can be configured from any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), a HDD (Hard Disk Drive), or an optical disk.

[0023] Some or all of the data (input data 121, true value data 122) stored in the storage unit 120 may be stored in physically different storage media, or may be stored in different storage areas of the same physically identical storage medium.

[0024] The display unit 131 is a display device for displaying various types of information, and is realized by, for example, a liquid crystal display, etc. Note that the display unit 131 does not need to be provided within the information processing device 100, and may be provided within an external device connected via a network (such as the Internet), for example.

[0025] The classification unit 101 executes a classification task (classification processing) using a machine learning model (first model). The machine learning model is a model that is trained to receive input data 121 and output, for each of a plurality of points (components), a classification class into which the points are classified from among a plurality of classification classes. For example, the classification unit 101 inputs input data for estimation (first input data) to the machine learning model and executes classification processing to obtain a classification class for each of a plurality of points (first components) output by the machine learning model.

[0026] The machine learning model may be any model that is trained to receive input data 121 and output classification results (classification classes) for each of a plurality of points. For example, the machine learning model may be realized by a neural network model. The method for constructing (training) the machine learning model may be any method that is applicable to the model to be adopted. Information (parameters, etc.) indicating the trained machine learning model may be stored in the storage unit 120.

[0027] The multiple classification classes may be any classes. For example, if the input data is 3D point cloud data for multiple points inside a building, the multiple classification classes may include a class indicating that the points are buildings and a class indicating that the points are equipment.

[0028] The reclassification unit 102 performs a reclassification process for each of a plurality of points included in the input data for estimation, to obtain a reclassification class into which the point is classified from a plurality of reclassification classes. For example, the reclassification unit 102 performs the reclassification process for each of a plurality of points, using the classification class obtained for the point by the classification unit 101 and additional information for the point. In this embodiment, the additional information is the true value of the classification class.

[0029] The plurality of reclassification classes include, for example, a class that indicates that the classification class matches the true value, and a class that indicates that the classification class does not match the true value, which are determined for each of the plurality of classification classes.

[0030] For example, if the multiple classification classes include class C_A and class C_B, the multiple reclassification classes may include four classes as follows: Class C_A' indicates that the estimated class C_A matches the true value. Class C_A'' indicates that the estimated class C_A does not match the true value. Class C_B' indicates that the estimated class C_B matches the true value. Class C_B'' indicates that the estimated class C_B does not match the true value.

[0031] The above four re-classification classes are examples in which each of the multiple classification classes includes two classes. The multiple re-classification classes may be set to include classes obtained by dividing at least some of the multiple classification classes.

[0032] The determination unit 103 determines the output mode of points classified into the re-classification classes for at least some of the re-classification classes. The output mode is, for example, color (hue, saturation, brightness), but may be any other mode such as whether or not to blink and the size of the points. The following mainly describes an example in which color is determined as the output mode.

[0033] The output control unit 104 controls the output of various information used in the information processing device 100. For example, the output control unit 104 controls the process of outputting (displaying) the reclassification results by the reclassification unit 102 to the display unit 131 in an output format determined by the determination unit 103.

[0034] When the input data is three-dimensional input data (such as three-dimensional point cloud data), the output control unit 104 controls the output of three-dimensional output data including, for example, element values ​​representing reclassified classes that are classification results for each three-dimensional position. Each point included in the three-dimensional output data is output in an output format (for example, color) determined by the determination unit 103.

[0035] At least a part of each of the above units (classification unit 101, reclassification unit 102, determination unit 103, and output control unit 104) may be realized by one or more processing units. Each of the above units is realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) execute a program, that is, by software. Each of the above units may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0036] The information processing device 100 may be physically configured as one device or may be physically configured as multiple devices. For example, the information processing device 100 may be built in a cloud environment. Furthermore, each unit within the information processing device 100 may be distributed across multiple devices.

[0037] Next, a description will be given of the visualization process performed by the information processing apparatus 100 according to the first embodiment. Fig. 2 is a flowchart showing an example of the visualization process according to the first embodiment.

[0038] The classification unit 101 inputs three-dimensional point cloud data (input data 121) into a machine learning model, and executes semantic segmentation to classify each point of the three-dimensional point cloud data into one of a plurality of classification classes (step S101).

[0039] The reclassification unit 102 compares the estimation result by the classification unit 101 with the true value (true value data 122) of the semantic segmentation for the input 3D point cloud data, and performs a reclassification process to classify each point of the 3D point cloud data into one of multiple reclassification classes based on the comparison result (step S102).

[0040] The reclassification criterion can be, for example, whether or not the estimated results match the true values. As an example, a case will be described in which three classification classes, class C_A, class C_B, and class C_C, are obtained as the estimated results of semantic segmentation. If the estimated results of class C_A, class C_B, and class C_C match the true values, the reclassification unit 102 reclassifies the corresponding points into class C_A', class C_B', and class C_C', respectively. If the estimated results of class C_A, class C_B, and class C_C do not match the true values, the reclassification unit 102 reclassifies the corresponding points into class C_A'', class C_B'', and class C_C'', respectively. In this example, the reclassification unit 102 reclassifies each point into one of six reclassification classes.

[0041] If the result does not match the true value, the reclassification class may be subdivided according to which classification class the misclassification was in. For example, if three classification classes, class C_A, class C_B, and class C_C, are obtained, the following reclassification classes may be used. If the true value is class C_A but is misclassified as class C_B: Class C_AB If the true value is class C_A but is misclassified as class C_C: class C_AC If the true value is class C_B but is misclassified as class C_A: Class C_BA If the true value is class C_B but is misclassified as class C_C: class C_BC If the true value is class C_C but is misclassified as class C_A: class C_CA If the true value is class C_C but is misclassified as class C_B: Class C_CB

[0042] In this case, the reclassification unit 102 will reclassify each point into one of nine reclassification classes, consisting of the three classes when the value matches the true value and the six classes mentioned above when the value does not match the true value.

[0043] Returning to the description of Fig. 2, the determination unit 103 determines a color for the reclassified class that is the result of the reclassification process (step S103).

[0044] For example, the determination unit 103 randomly assigns a color to each of the re-classified classes. Alternatively, the determination unit 103 may assign a pre-specified color to each of the re-classified classes.

[0045] The determination unit 103 may indirectly determine the colors of the reclassified classes by grouping some or all of the reclassified classes and assigning a color to each group. For example, the determination unit 103 classifies each of the reclassified classes into one of multiple groups and assigns a reference color representing a reference color to each of the multiple groups. The determination unit 103 determines a color obtained by varying at least a portion of the hue, saturation, and brightness of the reference color within a predetermined range as an output mode for each point classified into each of the multiple reclassified classes included in the multiple groups. The range of variation is predetermined, for example, so that the color does not overlap with the reference colors assigned to other groups. This color assignment method is effective when there are a large number of reclassified classes and it is difficult to manually assign colors to each reclassified class.

[0046] For example, the determination unit 103 classifies the re-classified classes into a correct group or an incorrect group. The correct group and the incorrect group are groups that include an accurate class and an incorrect class, respectively. The correct class is a class that indicates that the classification class obtained as the estimation result is correct, and in the above example, it is, for example, classes C_A', C_B', and C_C'. The incorrect class is a class that indicates that the classification class obtained as the estimation result is incorrect, and in the above example, it is, for example, classes C_A'', C_B'', and C_C''.

[0047] For example, the determination unit 103 assigns blue as the reference color to the correct answer group and red as the reference color to the incorrect answer group. The determination unit 103 determines a color obtained by varying the reference color blue for the reclassified classes included in the correct answer group (for example, classes C_A', C_B', C_C'). The determination unit 103 determines a color obtained by varying the reference color red for the reclassified classes included in the incorrect answer group (for example, classes C_A'', C_B'', C_C'').

[0048] Although the above example is an example in which all of the re-classified classes are grouped, the determination unit 103 may group only some of the re-classified classes. For example, the determination unit 103 creates only correct groups and classifies the correct classes into the correct groups. The determination unit 103 then assigns blue as a reference color to the correct groups. For the incorrect classes, the determination unit 103 determines a specified color (or a randomly selected color). For example, the determination unit 103 determines red, yellow, and orange for classes C_A'', C_B'', and C_C'', respectively.

[0049] Returning to the explanation of Fig. 2, the output control unit 104 displays each point using the determined color (step S104).

[0050] For example, the output control unit 104 colors each point of the three-dimensional point cloud data (input data 121) with a color assigned to the reclassification class to which the point belongs, and displays three-dimensional output data including the colored points on, for example, the display unit 131.

[0051] The output control unit 104 may color only points belonging to some re-classification classes with the color determined by the determination unit 103, and may use the color specified in the input three-dimensional point cloud data (input data 121) as is for points belonging to other re-classification classes. The output control unit 104 may color only points included in a specific region with the color determined by the determination unit 103, and may use the color specified in the input three-dimensional point cloud data (input data 121) as is for points included in other regions.

[0052] 3 is a diagram for explaining an example of visualization processing according to this embodiment. In FIG. 3, an example is shown in which three-dimensional point cloud data for each point inside a building is input as input data 121.

[0053] The area classification result 301 shows the result of the classification process on the input data 121. In the area classification result 301, gray points represent points classified into class C_A, which indicates that they are buildings, and white points represent points classified into class C_B, which indicates that they are facilities.

[0054] The reclassification unit 102 compares the region classification result 301 with the true value data 122 and reclassifies each point depending on whether they match (true or false). The reclassification result 302 represents the result of the reclassification process. Region 311 corresponds to a region containing points that do not match the true value for class C_A. Regions 321 and 322 correspond to regions containing points that do not match the true value for class C_B. For ease of explanation, FIG. 3 shows an example in which regions 311, 321, and 322 are distinguished by hatching, but they can also be configured to be distinguished by color instead of hatching.

[0055] 3, according to this embodiment, it is possible to simultaneously check the estimation result and additional information (information of true values) by displaying a single 3D point cloud data. In other words, it is possible to more easily check the estimation result by the machine learning model.

[0056] (Second embodiment) In the second embodiment, the reliability of the classification task is used as the additional information. Fig. 4 is a block diagram showing an example of the configuration of an information processing device 100-2 according to the second embodiment. As shown in Fig. 4, the information processing device 100-2 includes a storage unit 120-2, a display unit 131, a classification unit 101-2, a reclassification unit 102-2, a determination unit 103, and an output control unit 104.

[0057] In the second embodiment, the functions of the storage unit 120-2, the classification unit 101-2, and the reclassification unit 102-2 are different from those in the first embodiment. The other configurations and functions are the same as those in the block diagram of the information processing device 100 in the first embodiment shown in FIG. 1, and therefore the same reference numerals are used and the description thereof will be omitted here.

[0058] The storage unit 120-2 differs from the storage unit 120 of the first embodiment in that it does not store the true value data 122.

[0059] The classification unit 101-2 differs from the classification unit 101 of the first embodiment in that it executes classification processing so as to further output a reliability for each of a plurality of points. The reliability is an index indicating the degree to which a classification class, which is an estimation result of a classification task, is reliable. The reliability can also be interpreted as the reliability of classification by a machine learning model used by the classification unit 101-2.

[0060] The reliability may be calculated in any manner, but can be calculated, for example, by the following method. The score of the estimated result output by the machine learning model is used as the reliability. The entropy calculated from the above scores is used as the reliability. · Calculate statistical uncertainty as confidence level. - Use the confidence level output by the machine learning model as is.

[0061] The re-sorting unit 102-2 differs from the re-sorting unit 102 of the first embodiment in that it uses the reliability obtained by the sorting unit 101-2 as additional information.

[0062] Next, the visualization process performed by the information processing device 100-2 according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing an example of the visualization process according to the second embodiment.

[0063] The classification unit 101-2 inputs three-dimensional point cloud data (input data 121) into a machine learning model and performs semantic segmentation (step S201). In this embodiment, the classification unit 101-2 obtains the classification class of each point obtained by the semantic segmentation (classification process) and also obtains the reliability of the classification.

[0064] The reclassification unit 102-2 executes a reclassification process to classify each point of the three-dimensional point cloud data into one of a plurality of reclassification classes according to the reliability (step S202).

[0065] The criterion for reclassification can be, for example, whether the reliability of the estimation result is equal to or greater than a threshold. As an example, a case will be described in which three classification classes, class C_A, class C_B, and class C_C, are obtained as the estimation results of semantic segmentation. The reliability takes a value between 0 and 1. The threshold for the reliability is assumed to be 0.8.

[0066] If the reliability of the estimated results of classes C_A, C_B, and C_C is equal to or greater than a threshold value (0.8), the reclassification unit 102-2 reclassifies the corresponding points into classes C_A', C_B', and C_C', respectively. If the reliability of the estimated results of classes C_A, C_B, and C_C is less than a threshold value (0.8), the reclassification unit 102-2 reclassifies the corresponding points into classes C_A'', C_B'', and C_C'', respectively. In this example, the reclassification unit 102-2 reclassifies each point into one of the six reclassification classes.

[0067] The reclassification unit 102-2 may use multiple thresholds. In the following, an example will be described in which two thresholds, TH_1=0.4 and TH_2=0.8, are used. If the reliability of the estimated results of classes C_A, C_B, and C_C is equal to or greater than the threshold TH_2 (0.8), the reclassification unit 102-2 reclassifies the corresponding points into classes C_A', C_B', and C_C', respectively. If the reliability of the estimated results of classes C_A, C_B, and C_C is equal to or greater than the threshold TH_1 (0.4) and less than the threshold TH_2 (0.8), the reclassification unit 102-2 reclassifies the corresponding points into classes C_A'', C_B'', and C_C'', respectively. If the reliability of the estimated results of classes C_A, C_B, and C_C is less than threshold TH_1 (0.4), reclassification unit 102-2 reclassifies the corresponding points into class C_A''', class C_B''', and class C_C''', respectively. In this example, reclassification unit 102-2 reclassifies each point into one of nine reclassification classes.

[0068] The re-classification unit 102-2 may use a different threshold for each re-classification class. For example, the re-classification unit 102-2 may set the thresholds for class C_A, class C_B, and class C_C to 0.8, 0.5, and 0.0, respectively, and perform the same processing as above.

[0069] Returning to the description of Fig. 5, steps S203 to S204 are the same processes as steps S103 to S104 in the information processing apparatus 100 of the first embodiment, and therefore description thereof will be omitted.

[0070] In this way, the information processing apparatus of the second embodiment can achieve the same functions as those of the first embodiment by using the reliability of the classification process as additional information.

[0071] (Third embodiment) In the third embodiment, depth information of each point is used as additional information. Fig. 6 is a block diagram showing an example of the configuration of an information processing device 100-3 according to the third embodiment. As shown in Fig. 6, the information processing device 100-3 includes a storage unit 120-2, a display unit 131, a classification unit 101, a reclassification unit 102-3, a determination unit 103, an output control unit 104, and an estimation unit 105-3.

[0072] The third embodiment differs from the first embodiment in that the functions of a storage unit 120-2 and a reclassification unit 102-3, and an estimation unit 105-3 are added. The other configurations and functions are the same as those of the information processing device 100 of the first embodiment shown in FIG. 1, and therefore the same reference numerals are used and the description thereof will be omitted here.

[0073] The storage unit 120-2 is the same as in the second embodiment, so it is given the same reference numeral and the description thereof will be omitted here.

[0074] The re-sorting unit 102-3 differs from the re-sorting unit 102 of the first embodiment in that it uses depth information of each point as additional information.

[0075] The estimation unit 105-3 estimates depth information for each point. The depth information represents, for example, the distance of each point in the depth direction relative to a specified viewpoint. For example, the estimation unit 105-3 estimates the depth information using a machine learning model for depth estimation that is different from the machine learning model used for classification processing. The machine learning model for depth estimation is a model that is trained to input input data 121 and output depth information for each point included in the input data 121.

[0076] If three-dimensional point cloud data including depth information is available, the depth information included in the three-dimensional point cloud data may be used, in which case the estimation unit 105-3 may not be provided.

[0077] Next, visualization processing by the information processing device 100-3 according to the third embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing an example of visualization processing according to the third embodiment.

[0078] Step S301 is the same process as step S101 in the information processing device 100 of the first embodiment, and therefore a description thereof will be omitted.

[0079] In this embodiment, the estimation unit 105-3 inputs the 3D point cloud data (input data 121) into a machine learning model for depth estimation, and estimates the depth of each point in the 3D point cloud data (step S302). Note that the order of execution of steps S301 and S302 may be reversed, or they may be executed in parallel.

[0080] The reclassification unit 102-3 executes a reclassification process to classify each point of the three-dimensional point cloud data into one of a plurality of reclassification classes based on the depth information obtained in step S302 (step S303).

[0081] The reclassification criterion can be, for example, whether the depth obtained for each point for which an estimated result (classification class) is obtained is equal to or greater than a threshold. As an example, a case will be described in which three classification classes, class C_A, class C_B, and class C_C, are obtained as the estimated results of semantic segmentation. The depth threshold is assumed to be 10 m.

[0082] If the depths of points obtained as the estimation results of classes C_A, C_B, and C_C are equal to or greater than the threshold value (10 m), the reclassification unit 102-3 reclassifies the corresponding points into classes C_A', C_B', and C_C', respectively. If the depths of points obtained as the estimation results of classes C_A, C_B, and C_C are less than the threshold value (10 m), the reclassification unit 102-3 reclassifies the corresponding points into classes C_A'', C_B'', and C_C'', respectively. In this example, the reclassification unit 102-3 reclassifies each point into one of the six reclassification classes.

[0083] The re-sorting unit 102-3 may use a plurality of thresholds, similar to the re-sorting unit 102-2 of the second embodiment.

[0084] By reclassifying according to depth, it becomes possible to easily check, for example, the accuracy of classification processing for each depth range.

[0085] Returning to the description of Fig. 7, steps S304 to S305 are the same as steps S103 to S104 in the information processing apparatus 100 of the first embodiment, and therefore description thereof will be omitted.

[0086] In this way, the information processing device of the third embodiment can achieve the same functions as those of the first embodiment by using depth information as additional information.

[0087] (Example output) Next, examples of visualization processing of output data applicable to each of the above-described embodiments will be described with reference to FIGS. 8 to 12. FIGS. 8 to 12 are diagrams showing examples of display screens for displaying data. Note that, for ease of explanation, each figure shows an example in which the data output mode is distinguished by hatching, but it is also possible to configure the data to be distinguished by color instead of hatching. Also, for ease of explanation, each figure shows an example in which the point cloud for each reclassification class is schematically represented by a rectangular parallelepiped (FIGS. 8, 9, 10, and 12) or a planar figure (FIG. 11: circle, triangle, and square).

[0088] As shown in FIG. 8, the point cloud data to be visualized (visualized point cloud) is visualized in a different output format for each reclassified class (classes C_A', C_A'', C_B').

[0089] Data may be output in the determined output mode only for re-classification classes for which output in the determined output mode is specified. For example, the output control unit 104 controls the output of each of the multiple points in the determined output mode for re-classification classes for which output in the determined output mode is specified among the multiple re-classification classes.

[0090] Fig. 9 is a diagram showing an example of a display screen on which it is possible to specify whether or not to output (display / hide). Fig. 9 shows an example in which a check box 901 is deselected, so that output in the determined output mode is not specified for class C_B' among the re-classified classes. In this case, the point cloud 911 corresponding to class C_B' is displayed without being colored in the determined output mode (color). Point clouds of unspecified re-classified classes may be colored in the color specified in the input three-dimensional point cloud data (input data 121), or may be colored in a predetermined color (such as white or black) indicating that they are not displayed.

[0091] The specification of whether or not to output is not limited to a method using reclassification classes as units, and a method using classification classes that are the basis for reclassification as units may also be used. For example, the output control unit 104 controls the output of each of the multiple points in the determined output mode for a reclassification class that is reclassified based on a classification class for which output in the output mode is specified among the multiple classification classes.

[0092] Fig. 10 is a diagram showing an example of a display screen in this case. Fig. 10 shows an example in which a check box 1001 is selected, thereby specifying output in the determined output mode for classes C_A' and C_A'' that are reclassified based on class C_A among the classification classes. By making it possible to specify whether or not to output for each classification class, it becomes possible to specify whether or not to output for all reclassified classes at once.

[0093] So far, we have shown examples of visualizing classification results for a single input data set. The display screen may also include classification results for multiple input data sets, or classification results using multiple models for each of one or more input data sets.

[0094] Fig. 11 is a diagram showing an example of a display screen in this case. Fig. 11 shows an example of a display screen that displays, in a table format, the results of classification using three machine learning models (models M1, M2, M3) for each of three pieces of input data (data D1, D2, D3).

[0095] The multiple machine learning models are, for example, models that differ from one another in settings, parameters, structure, etc. The multiple machine learning models can be interpreted as corresponding to a certain machine learning model (first model) and one or more machine learning models (second models) that are different from the first machine learning model.

[0096] When multiple machine learning models are used, for example, the classification unit 101 performs classification processing using each of the multiple machine learning models. Furthermore, the reclassification unit 102 performs reclassification processing using classification classes obtained from each of the multiple machine learning models. The output control unit 104 outputs point clouds, which are classification results when multiple machine learning models are used, in a comparable manner, as shown in FIG. 11 .

[0097] By using a display screen that includes classification results from multiple machine learning models, it becomes possible to easily identify a machine learning model with higher classification accuracy, for example.

[0098] When the input data 121 is three-dimensional input data (three-dimensional point cloud data, three-dimensional mesh data, or three-dimensional image data), the output control unit 104 may rotatably display three-dimensional output data including the classification results obtained from the three-dimensional input data. For example, when rotation about at least some of the three-dimensional axes is specified, the output control unit 104 may output the three-dimensional output data while rotating it in accordance with the specification.

[0099] 12 is a diagram showing an example of a display screen in this case. The display screen in FIG. 12 includes an automatic rotation button 1201, axis selection buttons 1211, 1212, and 1213, and a slide bar 1221.

[0100] Axis selection buttons 1211, 1212, and 1213 correspond to the roll, pitch, and yaw axes, respectively. The axis corresponding to the pressed button is selected as the axis for rotating and displaying data when auto-rotate button 1201 is pressed. Slide bar 1221 can be used, for example, to adjust the speed of rotation.

[0101] As described above, according to the first to third embodiments, the estimation results obtained by the machine learning model can be more easily confirmed.

[0102] Next, the hardware configuration of the information processing apparatus according to the first to third embodiments will be described with reference to Fig. 13. Fig. 13 is an explanatory diagram showing an example of the hardware configuration of the information processing apparatus according to the first to third embodiments.

[0103] The information processing device of the first to third embodiments includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication I / F 54 that connects to a network and communicates, and a bus 61 that connects each part.

[0104] The programs executed by the information processing apparatuses of the first to third embodiments are provided in advance in the ROM 52 or the like.

[0105] The programs executed by the information processing devices of the first to third embodiments may be configured to be provided as a computer program product by being recorded in an installable or executable file format on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0106] Furthermore, the programs executed by the information processing apparatuses of the first to third embodiments may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Also, the programs executed by the information processing apparatuses of the first to third embodiments may be provided or distributed via a network such as the Internet.

[0107] The programs executed by the information processing devices of the first to third embodiments can cause a computer to function as each unit of the information processing device described above. In this computer, the CPU 51 can read the programs from a computer-readable storage medium onto a main storage device and execute the programs.

[0108] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0109] 100, 100-2, 100-3 Information processing device 101, 101-2 Classification section 102, 102-2, 102-3 Reclassification section 103 Decision Section 104 Output control section 105-3 Estimation section 120, 120-2 Storage section 121 Input Data 122 True Value Data 131 Display section

Claims

1. On the computer, a classification step of inputting first input data, which is input data for estimation, into a first model that is trained to input input data including a plurality of components and output, for each of the plurality of components, a classification class into which the components are classified out of a plurality of classification classes, and executing a classification process to obtain the classification class for each of a plurality of first components included in the first input data, which is output by the first model; a reclassification step of performing a reclassification process for obtaining, for each of the plurality of first components, a reclassification class into which the first component is classified from among a plurality of reclassification classes, using the classification class obtained for the first component and additional information of the first component; a determining step of determining, for at least some of the plurality of re-classification classes, an output mode of the first components classified into the re-classification classes; an output control step of controlling an output of the first component according to the determined output mode; A program to execute.

2. the additional information is either a true value of the classification class into which the first component is classified or a reliability of the classification process; The program according to claim 1.

3. the additional information is a truth value of the classification class into which the first component is classified, the plurality of re-classification classes include a class indicating that the classification class matches the true value and a class indicating that the classification class does not match the true value, which are determined for each of the plurality of classification classes; The program according to claim 1.

4. the additional information is a reliability of classification by the first model; The plurality of re-classification classes include a class indicating that the reliability is equal to or greater than a threshold and a class indicating that the reliability is less than the threshold, which are determined for each of the plurality of classification classes. The program according to claim 1.

5. the input data is three-dimensional input data including element values ​​for a plurality of the component elements, each of which corresponds to a three-dimensional position; The output control step includes: controlling output of three-dimensional output data including element values ​​representing the reclassification classes into which the first components for each of the three-dimensional positions are classified; The program according to claim 1.

6. The three-dimensional input data is three-dimensional point cloud data including element values ​​for each of a plurality of the components, each of which is a three-dimensional point; Three-dimensional mesh data including element values ​​for each of the plurality of components that are three-dimensional mesh-shaped regions, or The three-dimensional image data includes element values ​​for each of the plurality of components, which are three-dimensional pixels. The program according to claim 5.

7. the output control step, when rotation about at least a part of three-dimensional axes is designated, outputs the three-dimensional output data while rotating it in accordance with the designation. The program according to claim 5.

8. The input data is three-dimensional point cloud data including element values ​​that are positions of a plurality of the components that are a plurality of points inside the building. The program according to claim 1.

9. The determining step assigning a random color or a pre-designated color to each of the plurality of reclassification classes; determining a color assigned to the reclassification class into which the first component is classified as an output mode of the first component; The program according to claim 1.

10. The determining step assigning a reference color to each of a plurality of groups into which each of the plurality of reclassification classes is classified, and determining a color obtained by varying at least a part of the hue, saturation, and brightness from the reference color within a predetermined range as an output mode of the first component classified into each of the plurality of reclassification classes included in the plurality of groups; The program according to claim 1.

11. the plurality of re-classification classes include a correct class indicating that the classification class is correct and an incorrect class indicating that the classification class is incorrect, the correct class being determined for each of the plurality of classification classes; the plurality of groups include a correct group including the correct class determined for each of the plurality of classification classes, and an incorrect group including the incorrect class determined for each of the plurality of classification classes; The program according to claim 10.

12. The output control step includes: controlling the output of the first component in the determined output mode for the reclassification class for which output in the determined output mode is specified among the plurality of reclassification classes; The program according to claim 1.

13. The output control step includes: controlling the output of the first component in accordance with the determined output mode for the reclassified class reclassified based on the classification class for which output in accordance with the output mode is specified among the plurality of classification classes; The program according to claim 12.

14. the classification step performs the classification process using each of the first model and one or more second models different from the first model; the output control step outputs the first component when the first model is used and the first component when the second model is used so as to be comparable. The program according to claim 1.

15. the plurality of reclassified classes include classes obtained by dividing at least some of the plurality of classification classes; The program according to claim 1.

16. a classification unit that inputs first input data, which is input data for estimation, into a first model that is trained to receive input data including a plurality of components and output, for each of the plurality of components, a classification class into which the components are classified out of a plurality of classification classes, and executes a classification process to obtain the classification class for each of a plurality of first components included in the first input data, which is output by the first model; a re-classification unit that executes a re-classification process for obtaining, for each of a plurality of first components, a re-classification class into which the first component is classified from among a plurality of re-classification classes, using the classification class obtained for the first component and additional information of the first component; a determination unit that determines, for at least some of the re-classification classes, an output mode of the first components classified into the re-classification classes; an output control unit that controls the output of the first component according to the determined output mode; An information processing device comprising:

17. An information processing method executed by an information processing device, a classification step of inputting first input data, which is input data for estimation, into a first model that is trained to input input data including a plurality of components and output, for each of the plurality of components, a classification class into which the components are classified out of a plurality of classification classes, and executing a classification process to obtain the classification class for each of a plurality of first components included in the first input data, which is output by the first model; a reclassification step of performing a reclassification process for obtaining, for each of the plurality of first components, a reclassification class into which the first component is classified from among a plurality of reclassification classes, using the classification class obtained for the first component and additional information of the first component; a determining step of determining, for at least some of the plurality of re-classification classes, an output mode of the first components classified into the re-classification classes; an output control step of controlling an output of the first component according to the determined output mode; An information processing method including:

Images