Learning data generation device, learning device, inference device, and learning data generation method

Abstract

A learning data generation device (100) comprises: an image data acquisition unit (101) that acquires a plurality of pieces of image data; a feature amount extraction unit (102) that extracts a feature amount from each of the plurality of pieces of image data; a feature amount display unit (103) that causes a display device (10) to display the positions of the plurality of pieces of image data in a specific space according to the degrees of similarity between the feature amounts extracted by the feature amount extraction unit (102); a position information acquisition unit (104) that acquires position information indicating any position in the specific space selected by a user while the positions of the plurality of pieces of image data in the specific space are displayed on the display device (10); a data generation unit (105) that generates new data of a feature amount corresponding to the position indicated by the position information in the specific space; and a data reconstruction unit (106) reconstructs the new data generated by the data generation unit (105) into intermediate data of a feature amount, the number of dimensions of which is between the numbers of dimensions of the plurality of pieces of image data and the new data.

Description

Training data generation device, learning device, inference device, and training data generation method

[0001] The present disclosure relates to a training data generation device, a learning device, an inference device, and a training data generation method.

[0002] A training data generation device has been disclosed for increasing the variety and quantity of training data used to identify the class to which an image belongs without capturing new images (see Patent Document 1). This training data generation device is configured to detect low-density areas in which the density of feature values ​​is less than a predetermined density reference value in a feature space in which feature values ​​previously obtained from images belonging to each class to which an input image belongs are distributed, and to generate visible reconstructed images corresponding to sample points generated within the low-density areas, thereby increasing the variety and quantity of training data.

[0003] JP 2015-191426 A

[0004] However, in a device that performs machine learning using image data such as that described in Patent Document 1, depending on the amount of image data, the processing load on the device during the learning stage can be large, and there is a need to reduce the processing load on the device.

[0005] The present disclosure was made in recognition of the above-mentioned problems, and aims to provide a training data generation device, a learning device, an inference device, and a training data generation method that can reduce the processing burden on the device during the learning stage compared to conventional methods.

[0006] The training data generation device according to the present disclosure is characterized by comprising an image data acquisition unit that acquires multiple pieces of image data; a feature extraction unit that extracts features from each of the multiple pieces of image data; a feature display unit that displays, on a display device, positions of the multiple pieces of image data in a specific space according to the similarity of the respective features extracted by the feature extraction unit; a position information acquisition unit that acquires position information indicating any position in the specific space selected by a user while the positions of the multiple pieces of image data in the specific space are displayed on the display device; a data generation unit that generates new data of features according to the position indicated by the position information in the specific space; and a data reconstruction unit that reconstructs the new data generated by the data generation unit into intermediate data that is a feature with a number of dimensions between the number of pieces of image data and the new data.

[0007] The training data generation device according to the present disclosure can reduce the processing load on the device in the training stage compared to conventional devices.

[0008] 1 is a block diagram showing a schematic configuration of a learning system according to embodiment 1. FIG. 2 is a block diagram showing an example of the hardware configuration of a learning device according to embodiment 1. FIG. 3 is a block diagram showing an example of the hardware configuration of a learning device according to embodiment 1. FIG. 4 is a flowchart showing an example of processing performed by the learning device according to embodiment 1. FIG. 5 is a diagram showing positions in a feature space according to feature amounts of a plurality of image data that the learning device according to embodiment 1 causes to be displayed on a display device. FIG. 6 is a diagram showing positions in a feature space according to feature amounts of a plurality of image data and new data that the learning device according to embodiment 1 causes to be displayed on a display device. FIG. 7 is a flowchart showing an example of processing performed by an inference device according to embodiment 1. FIG. 8 is a flowchart showing an example of processing performed by a learning device according to embodiment 2. FIG. 9 is a block diagram showing a schematic configuration of a learning system according to embodiment 3. FIG. 10 is a flowchart showing an example of processing performed by the learning device according to embodiment 3.

[0009] Embodiments of the present disclosure will be described in detail below with reference to the drawings. Embodiment 1. First, a learning system 1 according to embodiment 1 will be described with reference to FIG. 1. FIG. 1 is a block diagram showing a schematic configuration of the learning system 1 according to embodiment 1. The learning system 1 shown in FIG. 1 is a system for inferring the probability of a class into which input image data is classified. For example, the learning system 1 detects a specific object, person, or the movement of an object or person contained as visual information in the input image data by inferring the probability of a class into which the input image data is classified. As shown in FIG. 1, the learning system 1 according to embodiment 1 includes a learning device 100, an inference device 500, a display device 10, and an input device 20, which are connected wirelessly or by wire to enable communication with each other. Note that it is sufficient for the inference device 500 to be able to acquire information from the learning device 100. The learning device 100 and the inference device 500 may be configured to be able to communicate information with each other via a device or communication line not shown, or may be integrated into a single device.

[0010] The display device 10 acquires information from the learning system 1 and displays the acquired information as visual information to the user of the learning system 1. For example, the display device 10 is configured by a liquid crystal display panel, an organic or inorganic EL (Electroluminescence) panel, a dot matrix display, or other device that displays information as visual information.

[0011] The input device 20 inputs information to the learning system 1. For example, the input device 20 accepts a user's input operation and inputs some or all of multiple image data previously acquired in response to the user's input operation into the learning system 1. Furthermore, for example, the input device 20 is configured with a keyboard, mouse, or other device, accepts the user's input operation, and generates a signal corresponding to the input operation. For example, the input device 20 may be configured as a touch panel that accepts a user's direct or indirect touch operation as a user's input operation and generates a signal corresponding to the position touched by the user, and may be formed integrally with the display device 10.

[0012] The learning device 100 is a device for generating a trained model for inferring the probability of a class into which input image data is classified. As shown in Fig. 1, the learning device 100 includes an image data acquisition unit 101, a feature extraction unit 102, a feature display unit 103, a position information acquisition unit 104, a data generation unit 105, a data reconstruction unit 106, and a learning unit 108.

[0013] The image data acquisition unit 101 acquires a plurality of image data for generating training data to be used in generating a trained model. For example, the image data acquisition unit 101 acquires a plurality of image data for generating training data to be used in generating a trained model based on information from the input device 20. Furthermore, for example, the image data acquisition unit 101 acquires a plurality of image data including image data labeled with class A and image data labeled with class B as the plurality of image data for generating training data.

[0014] The feature extraction unit 102 extracts feature amounts from each of the plurality of image data acquired by the image data acquisition unit 101. For example, the feature extraction unit 102 extracts feature amounts that are multidimensional vectors indicating brightness distribution, color distribution, edge arrangement, etc., for each image data acquired by the image data acquisition unit 101.

[0015] The feature display unit 103 outputs information indicating the results of processing by the learning device 100 to the display device 10, thereby displaying an image based on the information on the display device 10. For example, the feature display unit 103 causes the display device 10 to display the positions of the plurality of image data acquired by the image data acquisition unit 101 in a feature space, which is a virtual specific space, according to the similarity between the features extracted by the feature extraction unit 102. For example, the feature display unit 103 causes the display device 10 to display the positions of the plurality of image data acquired by the image data acquisition unit 101 in a feature space, which is a virtual two-dimensional space, according to the similarity between the features. This allows the user of the display device 10 to grasp the distribution in the feature space according to the similarity between the features of the plurality of image data acquired by the image data acquisition unit 101.

[0016] The position information acquisition unit 104 acquires position information indicating any position in the feature space selected by the user in a state in which the positions in the feature space of the plurality of image data acquired by the image data acquisition unit 101 are displayed on the display device 10. For example, in a state in which the positions in the feature space of the plurality of image data acquired by the image data acquisition unit 101 are displayed on the display device 10, when the user performs an operation on the input device 20 to select any position in the feature space, the position information acquisition unit 104 acquires, from the input device 20, position information indicating the position selected by the user.

[0017] The data generation unit 105 generates new data of feature quantities in the feature space according to the position indicated by the position information acquired by the position information acquisition unit 104. For example, the data generation unit 105 calculates two-dimensional feature quantities according to the position indicated by the position information acquired by the position information acquisition unit 104 based on the distance between the position indicated by the position information acquired by the position information acquisition unit 104 and the position of each feature quantity of the multiple image data in the feature space, and generates new data according to the calculated feature quantities. Furthermore, when generating new data, the data generation unit 105 assigns information indicating a class according to an input operation performed by the user on the input device 20 to the new data.

[0018] The data reconstruction unit 106 reconstructs the new data generated by the data generation unit 105 into intermediate data whose number of dimensions is a feature quantity between the number of dimensions of the new data and the plurality of image data acquired by the image data acquisition unit 101. In other words, the data reconstruction unit 106 converts the new data generated by the data generation unit 105 into intermediate data whose number of dimensions of the feature quantity is increased so that the number of dimensions becomes between the number of dimensions of the plurality of image data acquired by the image data acquisition unit 101 and the new data.

[0019] The learning unit 108 learns the multiple pieces of image data by inputting the feature quantities of the multiple pieces of image data extracted by the feature quantity extraction unit 102 into the input layer of the neural network, thereby generating a trained model. In other words, the learning unit 108 learns the relationship between the feature quantities of each piece of image data and the class of each piece of image data by inputting the feature quantities of the multiple pieces of image data extracted by the feature quantity extraction unit 102 into the input layer of the neural network, thereby generating a trained model that infers the probability of the class into which the input image data will be classified based on the input image data. The learning unit 108 also learns the intermediate data by inputting the feature quantities of the intermediate data generated by the data reconstruction unit 106 into the intermediate layer of the neural network, thereby generating a trained model. For example, the learning unit 108 outputs the generated trained model to the inference device 500.

[0020] In this way, the features of the multiple image data extracted by the feature extraction unit 102 and the intermediate data generated by the data reconstruction unit 106 are used as training data by the learning unit 108 to generate a trained model, and the image data acquisition unit 101, the feature extraction unit 102, the feature display unit 103, the position information acquisition unit 104, the data generation unit 105, and the data reconstruction unit 106 constitute the training data generation device in embodiment 1.

[0021] The inference device 500 is a device that infers the probability of a class into which input image data is classified, using a trained model generated by the learning device 100. As shown in Fig. 1 , the inference device 500 includes an image data acquisition unit 501, a feature extraction unit 502, an inference unit 503, and a storage unit 504.

[0022] The image data acquisition unit 501 acquires image data of an inference target to be inferred by the inference device 500. In other words, the image data acquisition unit 501 acquires image data of an inference target to be inferred using a trained model generated by the learning device 100. For example, the image data acquisition unit 501 acquires image data of an inference target to be inferred by the inference device 500 based on information from the input device 20. Note that the image data acquisition unit 501 constitutes an inference target image data acquisition unit in the first embodiment.

[0023] The feature extraction unit 502 extracts feature amounts from the image data acquired by the image data acquisition unit 501. For example, the feature extraction unit 502 extracts feature amounts that are multidimensional vectors indicating brightness distribution, color distribution, edge arrangement, etc. from the image data acquired by the image data acquisition unit 101.

[0024] The inference unit 503 uses the trained model generated by the learning device 100 to infer the class of the image data of the inference target acquired by the image data acquisition unit 501. In other words, the inference unit 503 uses the trained model generated by the learning device 100 to infer the probability of a class to which the image data of the inference target acquired by the image data acquisition unit 501 will be classified. Specifically, the inference unit 503 infers the class of the image data of the inference target acquired by the image data acquisition unit 501 by inputting the feature extracted by the feature extraction unit 502 into the trained model generated by the learning device 100. The inference unit 503 also outputs information indicating the inference result to the outside. For example, the inference unit 503 outputs the information indicating the inference result to the display device 10, thereby displaying an image based on the inference result on the display device 10. Specifically, the inference unit 503 outputs the information indicating the inference result to the display device 10, thereby displaying an image indicating the probability that the image data acquired by the image data acquisition unit 501 will be classified into each of a plurality of preset classes on the display device 10.

[0025] The storage unit 504 stores information used in various processes performed by the inference device 500 and information indicating the results of various processes performed by the inference device 500. For example, the storage unit 504 acquires from the learning device 100 and stores the trained model generated by the learning device 100. Note that in the first embodiment, the storage unit 504 constitutes a trained model acquisition unit that acquires the trained model. Each component of the inference device 500 refers to the information stored in the storage unit 504 when the inference device 500 performs processing.

[0026] Next, the hardware configuration of the learning device 100 will be described with reference to Figures 2 and 3. Figure 2 is a diagram showing an example of the hardware configuration of the learning device 100, and Figure 3 is a diagram showing an example of the hardware configuration of the learning device 100 that is different from that shown in Figure 2. For example, as shown in Figure 2, the learning device 100 is a computer having a processor 100a, a memory 100b, and an I / O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b.

[0027] 3, the learning device 100 is a computer that has a processing circuit 100d, which is dedicated hardware, and an I / O port 100c, and executes a program. The processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the learning device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program. Note that the learning device 100 may also have hardware other than those described above, such as a hardware timer.

[0028] The hardware configuration of the inference device 500 is similar to that of the learning device 100, and therefore a description thereof will be omitted.

[0029] Next, details of the processing performed by the learning device 100 will be described with reference to FIGS. 1 and 4 to 6. FIG. 4 is a flowchart illustrating an example of processing performed by the learning device 100 according to embodiment 1. The processing performed by the learning device 100 shown in FIG. 4 is processing for generating training data based on image data and generating a trained model for inferring the probability of classes into which input image data is classified. As shown in FIG. 4, when the learning device 100 starts processing, it first acquires image data (step ST01). In this processing, the learning device 100 acquires image data used to generate the training data using the image data acquisition unit 101.

[0030] After performing the process of step ST01, the learning device 100 extracts features from the image data (step ST02). In this process, the learning device 100 extracts features indicating the characteristics of the image data from the image data acquired in the process of step ST01 using the feature extraction unit 102.

[0031] After performing the process of step ST02, the learning device 100 performs learning of the image data using a neural network (step ST03). In this process, the learning device 100 inputs the feature quantities extracted in the process of step ST02 to the input layer of the neural network, thereby performing learning based on the image data acquired in the process of step ST01 using the learning unit 108.

[0032] After performing the process of step ST03, the learning device 100 arranges the image data in a feature space according to the features (step ST04). In this process, the learning device 100 calculates the position in the feature space of the image data acquired in the process of step ST01 based on the features extracted in the process of step ST02. For example, if there is image data whose position in the feature space has already been calculated, in this process, the learning device 100 calculates the position in the feature space of the image data acquired in the process of step ST01 based on the features extracted in the process of step ST02 according to the similarity with the features of other image data. Specifically, in this process, the learning device 100 calculates the position in the feature space of the image data acquired in the process of step ST01 based on the features extracted in the process of step ST02 so that the closer the similarity with the features of other image data is, the smaller the distance from the position of the other image data becomes.

[0033] After performing the processing of step ST04, the learning device 100 determines whether the acquisition of image data has been completed (step ST05). In this processing, the learning device 100 determines whether the acquisition of image data acquired by the image data acquisition unit 101 for generating learned data has been completed. If the acquisition of image data has not been completed in the processing of step ST05 (NO in step ST05), the learning device 100 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, thereby performing learning using multiple image data. Furthermore, for example, when the learning device 100 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, the learning device 100 updates the position of the feature of each image data in the feature space.

[0034] If the acquisition of image data has been completed in the process of step ST05 (YES in step ST05), learning device 100 displays the feature space in which the image data is arranged (step ST06). In this process, learning device 100 causes feature display unit 103 to display on display device 10 the positions in the feature space according to the degree of similarity between the features of the multiple image data acquired by repeatedly performing the processes of steps ST01 to ST05.

[0035] 5 is a diagram showing positions in a feature space corresponding to the features of multiple image data, which are displayed on the display device 10 by the learning device 100 according to embodiment 1. FIG. 5 shows positions in the feature space corresponding to the similarity of the features of multiple image data classified into class A and multiple image data classified into class B. For example, as shown in FIG. 5 , the multiple image data acquired in the processing of step ST01 is prone to erroneous classification of image data that should be classified into class A as class B, because the features of some image data classified into class A have high similarity with the features of image data classified into class B, but low similarity with the features of other image data classified into class A. In other words, as shown in FIG. 5 , the multiple image data acquired in the processing of step ST01 is prone to erroneous classification of image data that should be classified into class A as class B, because the features of some image data classified into class A and the features of other image data classified into class A are insufficient. In this way, the learning device 100 displays on the display device 10 the position in the feature space according to the degree of similarity between the features of multiple image data, thereby making it possible to visualize the likelihood of erroneous judgments due to the learning results and image data that is insufficient for learning.

[0036] After performing the processing of step ST06, the learning device 100 determines whether or not the user has selected a position in the feature space (step ST07). In this processing, the learning device 100 determines whether or not the user has selected any position in the feature space based on information from the input device 20, while positions in the feature space corresponding to the features of multiple image data are displayed on the display device 10.

[0037] If the user selects a position in the feature space in the processing of step ST07 (YES in step ST07), learning device 100 acquires position information corresponding to the position selected by the user (step ST08). In this processing, when learning device 100 acquires information from input device 20 indicating that the user has selected one of the positions in the feature space while positions in the feature space corresponding to the features of multiple image data are displayed on display device 10, learning device 100 acquires position information indicating the position in the feature space selected by the user based on the information acquired from input device 20.

[0038] 6 is a diagram showing positions in a feature space corresponding to the features of multiple image data and new data, which are displayed on the display device 10 by the learning device 100 according to embodiment 1. As shown in Figure 6, for example, while the positions in the feature space corresponding to the features of multiple image data are displayed on the display device 10, the learning device 100 uses the position information acquisition unit 104 to acquire position information indicating positions corresponding to the features of new data as additional samples, which has been selected by a user who has recognized that there is a shortage of data corresponding to the features between some image data classified into class A, which have features similar to those of image data classified into class B, and other image data classified into class A.

[0039] After performing the process of step ST08, learning device 100 generates new data according to the position information (step ST09). For example, in this process, while positions in the feature space corresponding to the features of the plurality of image data are displayed on display device 10, learning device 100 generates new data as additional samples having features calculated based on the distances between the position in the feature space selected by the user and each position corresponding to the feature of the plurality of image data acquired in the process of step ST01.

[0040] After completing step ST09, the learning device 100 reconstructs the new data into intermediate data with n-dimensional features (step ST13). In this process, the learning device 100 reconstructs, for example, the new data having two-dimensional features generated in step ST09 into intermediate data having n-dimensional features, which is a smaller number of dimensions than the multiple image data acquired in step ST01. This enables the learning device 100 to perform learning using the new data generated in step ST09 from the intermediate layer of the neural network, rather than the input layer. Note that n is, for example, a natural number greater than or equal to 3.

[0041] After performing the process of step ST13, the learning device 100 inputs the feature quantities of the reconstructed intermediate data into the intermediate layer of the neural network for learning (step ST17). In this process, the learning device 100 inputs the intermediate data reconstructed in the process of step ST13 into the intermediate layer of the neural network for learning, thereby generating a trained model based on new data added by user operation in addition to the multiple image data acquired in the process of step ST01.

[0042] After performing the process of step ST17, learning device 100 returns the process to step ST07 and determines whether the user has newly selected a position in the feature space. In this way, learning device 100 repeats the processes of steps ST07 to ST17, and generates one or more new data items corresponding to the positions selected by the user, depending on the number of times the user has selected a position in the feature space.

[0043] If the user does not select a position in the feature space in the processing of step ST07 (NO in step ST07), the learning device 100 outputs the trained model (step ST20). In this processing, the learning device 100 outputs the trained model trained based on multiple image data and new data added by user operation to an external device. For example, in this processing, the learning device 100 outputs the generated trained model to the storage unit 504 of the inference device 500, thereby storing the trained model in the storage unit 504.

[0044] Note that the learning device 100 is not limited to performing each process in the above-described order, and the above-described order of processes is merely one example of the processes performed by the learning device 100. For example, the learning device 100 may be configured to simultaneously perform learning on multiple image data and learning on new data, or may be configured to perform learning on multiple image data and learning on new data at different times rather than consecutively.

[0045] Next, details of the processing performed by the inference device 500 will be described with reference to FIGS. 1 and 7 . FIG. 7 is a flowchart illustrating an example of processing performed by the inference device 500 according to the first embodiment. The processing performed by the inference device 500 illustrated in FIG. 7 is processing for inferring the probability of a class into which input image data will be classified, using a trained model generated by the learning device 100. As illustrated in FIG. 4 , when the learning device 100 starts processing, it first acquires a trained model (step ST31). In this processing, the inference device 500 acquires the trained model generated by the learning device 100. For example, the inference device 500 refers to information stored in the storage unit 504 and acquires the trained model generated by the learning device 100 from the storage unit 504.

[0046] After performing the process of step ST31, the inference device 500 acquires image data (step ST32). In this process, the inference device 500 acquires, via the image data acquisition unit 501, image data of the object to be inferred using the trained model acquired in the process of step ST31.

[0047] After performing the process of step ST32, the inference device 500 extracts features from the image data (step ST33). In this process, the inference device 500 extracts features for performing inference using a trained model from the image data of the inference target acquired in the process of step ST32 using the feature extraction unit 502.

[0048] After completing the process of step ST33, the inference device 500 inputs the extracted feature amount into the trained model (step ST34). In this process, the inference device 500 infers, by the inference unit 53, the class into which the image data acquired in the process of step ST31 will be classified, based on the trained model generated by the learning device 100.

[0049] After performing the process of step ST34, the inference device 500 acquires an inference result (step ST35). In this process, the inference device 500 acquires, as an output of the trained model, the result of inference by the trained model in the process of step ST34, using the inference unit 53.

[0050] After performing the processing of step ST35, the inference device 500 outputs the inference result (step ST36). In this processing, the inference device 500 outputs the inference result of the trained model, which is the output from the trained model, to an external device. For example, in this processing, the inference device 500 outputs the probability of the class into which the image data acquired in the processing of step ST32 is classified as the inference result to the display device 10, and displays the inference result on the display device 10. Note that the inference device may be configured to output the inference result to the memory unit 504 and store it in the memory unit 504, or may be configured to output the inference result to another device (not shown) via a communication network (not shown).

[0051] As described above, the learning device 100 according to the first embodiment includes an image data acquisition unit 101 that acquires multiple pieces of image data, a feature extraction unit 102 that extracts features from each of the multiple pieces of image data, a feature display unit 103 that causes the display device 10 to display the positions of the multiple pieces of image data in a specific space according to the similarity of the mutual features extracted by the feature extraction unit 102, a position information acquisition unit 104 that acquires position information indicating any position in the specific space selected by the user while the positions of the multiple pieces of image data in the specific space are displayed on the display device 10, a data generation unit 105 that generates new data of features according to the position indicated by the position information in the specific space, and a data reconstruction unit 106 that reconstructs the new data generated by the data generation unit 105 into intermediate data whose number of dimensions is between that of the multiple image data and the new data.

[0052] For example, the learning device 100 according to the first embodiment includes a learning unit that learns multiple pieces of image data by inputting the features extracted by the feature extraction unit 102 into the input layer of the neural network, and learns new data by inputting the features of intermediate data into the intermediate layer of the neural network.

[0053] With this configuration, when learning using new data added by user operation, the learning device 100 can input intermediate data generated based on the new data into the intermediate layer of the neural network for learning, thereby reducing the processing burden on the device during the learning phase compared to conventional methods. Furthermore, the learning device 100 can visualize the distribution of the training data based on the similarity of the features of the training data by displaying on the display device 10 the positions in the feature space corresponding to the similarity of the features of the multiple image data used in the training data. This allows the user to visually grasp the features that are missing in the multiple image data used in the training data and that are necessary to improve the inference accuracy of the trained model. Furthermore, the learning device 100 allows the user to visually grasp the features that are missing in the multiple image data used in the training data and that are necessary to improve the inference accuracy of the trained model, thereby enabling the user to improve the inference accuracy of the trained model through user operation, even if the user does not have specialized knowledge.

[0054] In the first embodiment, the learning device 100 is configured to display positions in a feature space corresponding to the similarity of features of multiple image data used in the training data, but is not limited to this. The learning device may be configured to generate new data having features corresponding to any position in the feature space selected by the user, while positions in the feature space corresponding to the similarity of multiple image data are displayed on the display device 10. For example, the learning device may be configured to generate new data having features corresponding to any position in the feature space selected by the user, while positions in the feature space corresponding to the features of multiple image data classified by the trained model are displayed on the display device 10.

[0055] In the first embodiment, the learning device 100 is configured to display on the display device 10 the positions in the feature space, which is a two-dimensional space, of the plurality of image data acquired by the image data acquisition unit 101, but is not limited to this. The learning device only needs to be configured to display on the display device 10 the respective positions in the feature space corresponding to the similarity of the plurality of image data acquired by the image data acquisition unit 101 in a manner that is visually graspable by the user. As long as the manner is visually graspable by the user, the learning device may be configured to display on the display device 10 the positions in the feature space, which is a three-dimensional space, of the plurality of image data, or may be configured to display in the feature space in a manner corresponding to the similarity of four or more dimensions of features.

[0056] 1 and 8, a learning system 2 according to embodiment 2 will be described. The learning system 2 according to embodiment 2 differs from the learning system 1 according to embodiment 1 in the configuration of some of the learning devices, but the other configurations are the same. The same components as those in embodiment 1 are given the same names and symbols as those in embodiment 1, and their description will be omitted.

[0057] As shown in FIG. 1 , a learning system 2 according to the second embodiment includes a learning device 200, an inference device 500, a display device 10, and an input device 20, which are connected wirelessly or by wire to enable communication with each other. The learning device 200 is a device that generates a trained model for inferring the probability of a class into which input image data is classified. Note that the inference device 500 only needs to be able to acquire information from the learning device 200. The learning device 200 and the inference device 500 may be configured to be able to communicate information with each other via a device or communication line (not shown), or may be configured integrally as a single device.

[0058] The learning device 200 includes an image data acquisition unit 101, a feature extraction unit 102, a feature display unit 203, a position information acquisition unit 104, a data generation unit 105, a data reconstruction unit 206, and a learning unit 108.

[0059] The data reconstruction unit 206 reconstructs the new data generated by the data generation unit 105 into image data. In other words, the data reconstruction unit 206 increases the number of dimensions of the feature quantities of the new data having two-dimensional feature quantities generated by the data generation unit 105 to a number of dimensions that can be visually displayed as image data. For example, the data reconstruction unit 206 reconstructs the new data into image data based on the method described in Patent Document 1 for generating a visually reconstructed image corresponding to the sample points. Note that the data reconstruction unit 206 has all of the other functions possessed by the data reconstruction unit 106 according to the first embodiment, and the other functions possessed by the data reconstruction unit 106 are the same as those in the first embodiment, so description thereof will be omitted.

[0060] The feature display unit 203 displays, on the display device 10, image data obtained by reconstructing the new data generated by the data generation unit 105 using the data reconstruction unit 206. In other words, the feature display unit 203 displays, on the display device 10, image data obtained by reconstructing the new data generated by the data generation unit 105 using the data reconstruction unit 206. In this way, the feature display unit 203 enables the user to determine whether the image data indicating the feature amounts of the new data generated by the user's operation is appropriate as learning data. Note that the feature display unit 203 has all of the other functions possessed by the feature display unit 103 according to the first embodiment, and the other functions possessed by the feature display unit 103 are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0061] The hardware configuration of the learning device 200 is similar to that of the learning device 100, and therefore a description thereof will be omitted.

[0062] Next, details of the processing performed by the learning device 200 will be described with reference to Figures 1 and 8. Figure 8 is a flowchart showing an example of processing performed by the learning device 200 according to embodiment 2. The processing performed by the learning device 200 shown in Figure 8 is processing for generating training data based on image data and generating a trained model for inferring the probability of a class into which input image data is classified. Note that part of the processing performed by the learning device 200 according to embodiment 2 is similar to the processing performed by the learning device 100 according to embodiment 1, and therefore, a description of processing similar to that of embodiment 1 will be omitted.

[0063] As shown in FIG. 8 , when the learning device 200 starts processing, it first acquires image data (step ST01). After performing the processing of step ST01, the learning device 200 extracts features from the image data (step ST02). After performing the processing of step ST02, the learning device 200 trains the image data using a neural network (step ST03). After performing the processing of step ST03, the learning device 200 arranges the image data in a feature space according to the features (step ST04). After performing the processing of step ST04, the learning device 200 determines whether or not image data acquisition has been completed (step ST05). If image data acquisition has not been completed in the processing of step ST05 (NO in step ST05), the learning device 200 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, thereby performing learning using multiple image data. Furthermore, for example, when the learning device 200 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, the learning device 200 updates the position of the features of each image data in the feature space.

[0064] If the acquisition of image data is completed in the processing of step ST05 (YES in step ST05), the learning device 200 displays the feature space in which the image data is arranged (step ST06). After performing the processing of step ST06, the learning device 200 determines whether or not the user has selected a position in the feature space (step ST07). If the user has selected a position in the feature space in the processing of step ST07 (YES in step ST07), the learning device 200 acquires position information corresponding to the position selected by the user (step ST08). After performing the processing of step ST08, the learning device 200 generates new data according to the position information (step ST09).

[0065] After completing the process of step ST09, learning device 200 reconstructs the new data into image data (step ST10). In this process, learning device 200 increases the number of dimensions of the features of the new data having two-dimensional features generated in the process of step ST09 to a number of dimensions that can be visually displayed as image data.

[0066] After completing the process of step ST10, learning device 200 displays the reconstructed image data (step ST11). In this process, learning device 200 outputs the image data reconstructed in the process of step ST10 to display device 10, thereby displaying the image data on display device 10.

[0067] After performing the process of step ST11, learning device 200 determines whether the reconstructed image data is suitable for learning (step ST12). In this process, learning device 200 displays the image data generated based on the new data in the process of step ST11 on display device 10, and acquires information from input device 20 indicating that the user has determined whether the image data is suitable for use in learning, thereby determining whether the image data should be treated as suitable for learning.

[0068] If the reconstructed image data is determined to be suitable for learning in the process of step ST12 (YES in step ST12), the learning device 200 reconstructs the new data into intermediate data, which are n-dimensional features (step ST13). After performing the process of step ST13, the learning device 200 inputs the features of the reconstructed intermediate data into the intermediate layer of the neural network for learning (step ST17). If the reconstructed image data is determined to be unsuitable for learning in the process of step ST12 (NO in step ST12) and after performing the process of step ST17, the learning device 200 returns to the process of step ST07 and determines whether the user has selected a new position in the feature space.

[0069] If the user has not selected a position in the feature space in the processing of step ST07 (NO in step ST07), the learning device 200 outputs the learned model (step ST20).

[0070] As described above, the learning device 200 according to the second embodiment is configured to reconstruct new data into image data, and the feature display unit is configured to display the image data reconstructed from the new data on the display device. With this configuration, the learning device 200 enables the user to visually determine whether new data generated by the user's operation is suitable for use as training data, and can generate training data that improves the accuracy of inference using a trained model.

[0071] 9 and 10, a learning system 3 according to embodiment 3 will be described. The learning system 3 according to embodiment 3 differs from the learning system 1 according to embodiment 1 in the configuration of some of the learning devices, but the other configurations are the same. The same components as those in embodiment 1 are given the same names and symbols as those in embodiment 1, and their description will be omitted.

[0072] As shown in FIG. 1 , a learning system 2 according to the third embodiment includes a learning device 300, an inference device 500, a display device 10, and an input device 20, which are connected wirelessly or by wire to enable communication with each other. The learning device 300 is a device that generates a trained model for inferring the probability of a class into which input image data is classified. Note that the inference device 500 only needs to be able to acquire information from the learning device 300. The learning device 300 and the inference device 500 may be configured to be able to communicate information with each other via a device or communication line (not shown), or may be configured integrally as a single device.

[0073] The learning device 300 includes an image data acquisition unit 101, a feature extraction unit 302, a feature display unit 103, a position information acquisition unit 104, a data generation unit 105, a data reconstruction unit 106, a feature comparison unit 107, and a learning unit 108.

[0074] The feature extraction unit 302 extracts feature amounts with a number of dimensions corresponding to the feature amounts of the intermediate data reconstructed by the data reconstruction unit 106 from each of the multiple image data acquired by the image data acquisition unit 101. Note that the feature extraction unit 302 has all the other functions of the feature extraction unit 102 according to the first embodiment, and the other functions of the feature extraction unit 102 are the same as those of the first embodiment, so description thereof will be omitted.

[0075] The feature comparison unit 107 compares the feature amounts of the multiple pieces of image data extracted by the feature extraction unit 302 with the feature amounts of the intermediate data and calculates the similarity between the respective feature amounts. For example, the feature comparison unit 107 compares the feature amounts of the multiple pieces of image data extracted by the feature extraction unit 302 with the feature amounts of the intermediate data and calculates the distance between the respective feature amounts as the similarity between the respective feature amounts. Furthermore, the feature comparison unit 107 selects one of the multiple pieces of image data acquired by the image data acquisition unit 101 based on the calculation result of the similarity between the feature amounts of the multiple pieces of image data extracted by the feature extraction unit 302 and the feature amounts of the intermediate data. For example, the feature comparison unit 107 selects the image data with the highest similarity from the multiple pieces of image data acquired by the image data acquisition unit 101 based on the calculation result of the similarity between the feature amounts of the multiple pieces of image data extracted by the feature extraction unit 302 and the feature amounts of the intermediate data.

[0076] Based on the comparison result by the feature comparison unit 107, the feature display unit 103 displays one of the plurality of image data acquired by the image data acquisition unit 101 on the display device 10. For example, based on the comparison result by the feature comparison unit 107, the feature display unit 103 outputs information about the image data among the plurality of image data acquired by the image data acquisition unit 101 that has the highest feature similarity to the intermediate data generated by the user's operation to the display device 10, and displays the image data on the display device 10. In this way, the feature display unit 103 allows the user to determine whether the intermediate data generated by the user's operation is appropriate as learning data.

[0077] The hardware configuration of the learning device 300 is similar to that of the learning device 100, and therefore a description thereof will be omitted.

[0078] Next, details of the processing performed by the learning device 300 will be described with reference to Figures 9 and 10. Figure 10 is a flowchart illustrating an example of processing performed by the learning device 300 according to embodiment 3. The processing performed by the learning device 300 shown in Figure 10 is processing for generating training data based on image data and generating a trained model for inferring the probability of a class into which input image data is classified. Note that part of the processing performed by the learning device 300 according to embodiment 3 is similar to the processing performed by the learning device 100 according to embodiment 1 and the processing performed by the learning device 200 according to embodiment 2, and therefore, a description of processing similar to that of embodiments 1 and 2 will be omitted.

[0079] As shown in FIG. 8 , when the learning device 300 starts processing, it first acquires image data (step ST01). After performing the processing of step ST01, the learning device 300 extracts features from the image data (step ST02). After performing the processing of step ST02, the learning device 300 trains the image data using a neural network (step ST03). After performing the processing of step ST03, the learning device 300 arranges the image data in a feature space according to the features (step ST04). After performing the processing of step ST04, the learning device 300 determines whether or not image data acquisition has been completed (step ST05). If image data acquisition has not been completed in the processing of step ST05 (NO in step ST05), the learning device 300 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, thereby performing learning using multiple image data. Furthermore, for example, when the learning device 300 returns to the processing of step ST01 and repeats the processing of steps ST01 to ST05, the learning device 300 updates the position of the features of each image data in the feature space.

[0080] If image data acquisition is completed in step ST05 (YES in step ST05), the learning device 300 displays the feature space in which the image data is arranged (step ST06). After performing step ST06, the learning device 300 determines whether the user has selected a position in the feature space (step ST07). If the user has selected a position in the feature space in step ST07 (YES in step ST07), the learning device 300 acquires position information corresponding to the position selected by the user (step ST08). After performing step ST08, the learning device 300 generates new data according to the position information (step ST09). After performing step ST09, the learning device 300 reconstructs the new data into intermediate data, which are n-dimensional features (step ST13).

[0081] After performing the process of step ST13, the learning device 300 extracts n-dimensional features from the image data (step ST14). In this process, the learning device 300 uses the feature extraction unit 302 to extract n-dimensional features, which is the number of dimensions corresponding to the intermediate data generated in the process of step ST13, so as to make it possible to compare the multiple image data acquired in the process of step ST01 with the intermediate data.

[0082] After performing the process of step ST14, the learning device 300 selects image data that is closest to the feature amounts of the intermediate data (step ST15). In this process, the learning device 300 compares the feature amounts of the multiple image data extracted in the process of step ST14 with the feature amounts of the intermediate data, and selects, by the feature amount comparison unit 107, the image data that has the highest similarity between the feature amount extracted in the process of step ST14 and the feature amount of the intermediate data from the multiple image data acquired in the process of step ST01.

[0083] After completing the process of step ST15, the learning device 300 displays the image data that is closest to the feature values ​​of the intermediate data (step ST16). In this process, the learning device 300 displays on the display device 10 the image data that has the highest similarity between the feature values ​​extracted in the process of step ST14 and the feature values ​​of the intermediate data, thereby allowing the user to determine whether the intermediate data is suitable as learning data.

[0084] After performing the processing of step ST16, the learning device 300 determines whether the reconstructed image data is suitable for learning (step ST12). If the processing of step ST12 determines that the reconstructed image data is suitable for learning (YES in step ST12), the learning device 300 inputs the features of the reconstructed intermediate data into the intermediate layer of the neural network and performs learning (step ST17). If the processing of step ST12 determines that the reconstructed image data is not suitable for learning (NO in step ST12) and if the processing of step ST17 is performed, the learning device 300 returns to the processing of step ST07 and determines whether a new position in the feature space has been selected by the user. If the processing of step ST07 does not determine that a position in the feature space has been selected by the user (NO in step ST07), the learning device 300 outputs the trained model (step ST20).

[0085] As described above, the learning device 300 according to the third embodiment is configured to extract, from each of a plurality of image data, feature quantities with a number of dimensions corresponding to the feature quantities of the intermediate data, and to display one of the plurality of image data on the display device 10 based on the similarity between the feature quantities of the dimensions corresponding to the feature quantities of the intermediate data and the feature quantities of the intermediate data. Configured in this way, the learning device 300 enables the user to visually determine whether new data generated by user operation is suitable for use as learning data, and can generate learning data for improving the accuracy of inference by a trained model.

[0086] In the third embodiment, the learning device 300 is configured to display on the display device 10, among the plurality of image data acquired by the image data acquisition unit 101, image data having the highest feature similarity to the intermediate data generated by a user operation. However, this is not limiting. The learning device may be configured to display on the display device 10 any of the plurality of image data based on the similarity between the dimensional feature corresponding to the feature of the intermediate data and the feature of the intermediate data. For example, the learning device may be configured to display on the display device 10 a plurality of image data whose similarity is equal to or greater than a threshold based on the similarity between the dimensional feature corresponding to the feature of the intermediate data and the feature of the intermediate data, or may be configured to display on the display device 10 a specific number of image data whose similarity is highest based on the similarity between the dimensional feature corresponding to the feature of the intermediate data and the feature of the intermediate data.

[0087] The learning device may have some or all of the configuration of the inference device, or the inference device may have some or all of the configuration of the learning device, or the learning device or some of the configuration of the inference device may be provided in another device that is communicatively connected to the learning device and the inference device. Also, some of the configuration of the learning device may have some of the functions of the inference device, or some of the configuration of the inference device may have some of the functions of the learning device.

[0088] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0089] An information processing device according to the present disclosure can be used, for example, to reduce the processing load on a learning device during the learning stage when generating a trained model that detects objects based on image data.

[0090] Various aspects of the present disclosure are summarized below as appendices.

[0091] (Supplementary Note 1) A training data generation device comprising: an image data acquisition unit that acquires a plurality of image data; a feature extraction unit that extracts features from each of the plurality of image data; a feature display unit that causes a display device to display positions of the plurality of image data in a specific space corresponding to the similarity of the mutual features extracted by the feature extraction unit; a position information acquisition unit that acquires position information indicating any position in the specific space selected by a user while the positions of the plurality of image data in the specific space are displayed on the display device; a data generation unit that generates new data of feature quantities corresponding to the position indicated by the position information in the specific space; and a data reconstruction unit that reconstructs the new data generated by the data generation unit into intermediate data whose number of dimensions is between that of the plurality of image data and the new data. (Supplementary Note 2) The training data generation device according to Supplementary Note 1, wherein the data reconstruction unit reconstructs the new data into image data, and the feature display unit causes the display device to display the image data from which the new data has been reconstructed. (Supplementary Note 3) The training data generation device according to Supplementary Note 1 or 2, wherein the feature extraction unit extracts features of a number of dimensions corresponding to the features of the intermediate data from each of the plurality of image data, and the feature display unit displays one of the plurality of image data on the display device based on a similarity between a feature of a dimension corresponding to the feature of the intermediate data and the feature of the intermediate data. (Supplementary Note 4) A training data generation device comprising: the training data generation device according to any one of Supplementary Notes 1 to 3; and a learning unit that learns the plurality of image data by inputting the features extracted by the feature extraction unit to an input layer of a neural network, wherein the learning unit learns the new data by inputting the features of the intermediate data to an intermediate layer of the neural network.(Supplementary Note 5) An inference device comprising: a trained model acquisition unit that acquires a trained model generated by learning the new data by inputting feature values ​​of a plurality of image data into an input layer of a neural network, and while displaying positions in a specific space corresponding to the similarity of the feature values ​​of the plurality of image data on a display device, generates new data for feature values ​​corresponding to any position in the specific space selected by a user relative to the positions of the plurality of image data, reconstructs the generated new data into intermediate data whose number of dimensions is between the number of dimensions of the plurality of image data and the new data, and inputs the feature values ​​of the intermediate data into an intermediate layer of the neural network; an inference target image data acquisition unit that acquires image data to be inferred; and an inference unit that infers the class of the image data acquired by the inference target image data acquisition unit using the trained model. (Supplementary Note 6) A training data generation method performed by an apparatus including an image data acquisition unit, a feature extraction unit, a feature display unit, a position information acquisition unit, a data generation unit, and a data reconstruction unit, the training data generation method comprising: a step in which the image data acquisition unit acquires a plurality of image data; a step in which the feature extraction unit extracts features from each of the plurality of image data; a step in which the feature display unit displays, on a display device, positions of the plurality of image data in a specific space according to the similarity between the respective features extracted by the feature extraction unit; a step in which the position information acquisition unit acquires position information indicating any position in the specific space selected by a user, with the positions of the plurality of image data in the specific space being displayed on a display device; a step in which the data generation unit generates new data in the specific space having features according to the position indicated by the position information; and a step in which the data reconstruction unit reconstructs the new data generated by the data generation unit into intermediate data whose number of dimensions is between that of the plurality of image data and the new data.

[0092] 1 Learning system, 2 Learning system, 3 Learning system, 10 Display device, 20 Input device, 53 Inference unit, 100 Learning device (learning data generation device), 100a Processor, 100b Memory, 100c I / O port, 100d Processing circuit, 101 Image data acquisition unit, 102 Feature extraction unit, 103 Feature display unit, 104 Position information acquisition unit, 105 Data generation unit, 106 Data reconstruction unit, 107 Feature comparison unit, 108 Learning unit, 200 Learning device, 203 Feature display unit, 206 Data reconstruction unit, 300 Learning device, 302 Feature extraction unit, 500 Inference device, 501 Image data acquisition unit, 502 Feature extraction unit, 503 Inference unit, 504 Memory unit.

Claims

1. A training data generation device comprising: an image data acquisition unit that acquires multiple pieces of image data; a feature extraction unit that extracts features from each of the multiple pieces of image data; a feature display unit that displays on a display device the positions of the multiple pieces of image data in a specific space according to the similarity of the mutual features extracted by the feature extraction unit; a position information acquisition unit that acquires position information indicating any position in the specific space selected by a user while the positions of the multiple pieces of image data in the specific space are displayed on the display device; a data generation unit that generates new data of features according to the position indicated by the position information in the specific space; and a data reconstruction unit that reconstructs the new data generated by the data generation unit into intermediate data whose number of dimensions is between that of the multiple pieces of image data and the new data.

2. The training data generation device according to claim 1, characterized in that the data reconstruction unit reconstructs the new data into image data, and the feature display unit displays the image data into which the new data has been reconstructed on the display device.

3. The training data generation device according to claim 1 or 2, characterized in that the feature extraction unit extracts features of a number of dimensions corresponding to the features of the intermediate data from each of the plurality of image data, and the feature display unit displays one of the plurality of image data on the display device based on the similarity between the features of the dimensions corresponding to the features of the intermediate data and the features of the intermediate data.

4. A learning device comprising: a learning data generation device according to any one of claims 1 to 3; and a learning unit that learns the plurality of image data by inputting the features extracted by the feature extraction unit into an input layer of a neural network, wherein the learning unit learns the new data by inputting the features of the intermediate data into an intermediate layer of the neural network.

5. An inference device comprising: a trained model acquisition unit that acquires a trained model generated by learning the new data by inputting feature values ​​of multiple image data into an input layer of a neural network, and while displaying positions in a specific space corresponding to the similarity of the feature values ​​of the multiple image data on a display device, generates new data for feature values ​​corresponding to any position in the specific space selected by a user relative to the positions of the multiple image data, reconstructs the generated new data into intermediate data whose number of dimensions is between the number of dimensions of the multiple image data and the new data, and inputs the feature values ​​of the intermediate data into an intermediate layer of the neural network; an inference target image data acquisition unit that acquires image data to be inferred; and an inference unit that infers the class of the image data acquired by the inference target image data acquisition unit using the trained model.

6. A training data generation method performed by an apparatus including an image data acquisition unit, a feature extraction unit, a feature display unit, a position information acquisition unit, a data generation unit, and a data reconstruction unit, comprising: a step in which the image data acquisition unit acquires multiple pieces of image data; a step in which the feature extraction unit extracts features from each of the multiple pieces of image data; a step in which the feature display unit displays, on a display device, positions of the multiple pieces of image data in a specific space according to the similarity between the features extracted by the feature extraction unit; a step in which the position information acquisition unit acquires position information indicating any position in the specific space selected by a user, with the positions of the multiple pieces of image data in the specific space being displayed on a display device; a step in which the data generation unit generates new data in the specific space having features according to the position indicated by the position information; and a step in which the data reconstruction unit reconstructs the new data generated by the data generation unit into intermediate data whose number of dimensions is between that of the multiple pieces of image data and the new data.