Data creation device

Abstract

To provide a data generation apparatus and a learning system capable of easily generating a large amount of training data by which object recognition or voice recognition can be properly executed.SOLUTION: A data generation apparatus 1 for generating training data for machine learning includes: a data input unit which receives input of real data on a target obtained from a physical space and virtual data obtained by modeling the target from a virtual space; a noise information acquisition unit which acquires, for the same target, noise information included in the real data but not included in the virtual data; and a training data generation unit which generates training data of a recognition target by synthesizing the noise information with recognition target virtual data obtained by modeling the recognition target from the virtual space.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a data creation device for creating teacher data for machine learning.

Background Art

[0002] In recent years, the practical application of object recognition and speech recognition by machine learning such as deep learning has been promoted. In order to improve the accuracy of object recognition and speech recognition by machine learning, a large amount of teacher data is required, so teacher data is augmented by data augmentation. Patent Document 1 discloses an example of a data augmentation system.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when augmenting teacher data, it takes a lot of time and effort to collect a large amount of real data (data obtained by photographing or recording real objects, etc.) obtained from the physical space. Therefore, it is not easy to create a large amount of teacher data by using only real data as teacher data.

[0005] Therefore, it is conceivable to use virtual data (data obtained by modeling objects, sounds, etc.) as teacher data. Thereby, compared with the case of using only real data as teacher data, a large amount of teacher data can be easily created. Also, if virtual data is used as teacher data, for example, when creating teacher data for an article that is still in the design stage in advance, it is possible to create teacher data without creating a prototype (real object).

[0006] However, real data obtained from photographing or recording real objects may contain imperfections such as halation and occlusion, whereas virtual data does not. Because of this gap between real and virtual data, using virtual data as training data for machine learning can lead to problems such as inaccurate object recognition and speech recognition.

[0007] In view of the above-mentioned problems, the present invention aims to provide a data creation device that can easily create a large amount of training data that enables appropriate execution of object recognition, speech recognition, and the like. [Means for solving the problem]

[0008] The data creation device according to the present invention is a data creation device for creating training data for machine learning, and comprises: a data input unit that accepts input of real data of an object obtained from physical space and virtual data obtained from virtual space by modeling an object; a noise information acquisition unit that acquires noise information for the same object that is included in the real data but not in the virtual data; and a training data creation unit that synthesizes the noise information with virtual data of a recognition target obtained from virtual space by modeling a recognition target to create training data for the recognition target.

[0009] This configuration makes it easy to create a large amount of training data that enables proper execution of object recognition, speech recognition, etc. Here, "creating training data by synthesizing noise information with virtual data to be recognized" includes the concepts of "creating training data by synthesizing noise information with virtual data to be recognized that has undergone transformation processing," and "creating training data by further transformation processing on virtual data to be recognized that has had noise information synthesized with it." Furthermore, machine learning here is a concept that includes machine learning for object recognition and machine learning for speech recognition; in the former case, "target" and "recognition target" are objects, and in the latter case, "target" and "recognition target" are sounds. "Real data of the target obtained from physical space" is a concept that includes real data obtained by photographing or recording the target. [Effects of the Invention]

[0010] According to the data creation device of the present invention, it is possible to easily create a large amount of training data that enables object recognition, speech recognition, and the like to be performed appropriately. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the schematic configuration of the data creation device 1 of this embodiment. [Figure 2] This is a flowchart illustrating the noise information acquisition process. [Figure 3] This is an explanatory diagram regarding the operation of acquiring noise information. [Figure 4] This is an explanatory diagram regarding the operation of acquiring noise information. [Figure 5] This is a flowchart illustrating the process of creating training data. [Figure 6] This is an explanatory diagram of the first pattern of method for creating training data. [Figure 7] This is an explanatory diagram of the second pattern of method for creating training data. [Figure 8] This is an explanatory diagram of the third pattern of method for creating training data. [Figure 9] This is a flowchart illustrating the process of creating noise information. [Figure 10] This is an explanatory diagram regarding the noise information creation process. [Modes for carrying out the invention]

[0012] One embodiment of the present invention will be described below with reference to the drawings.

[0013] 1. Configuration of the data creation device, etc. Figure 1 is a block diagram showing the schematic configuration of the data creation device 1 according to this embodiment. The data creation device 1 is a device for creating training data for deep learning, and comprises a data input unit 11, a noise information acquisition unit 12, a noise information storage unit 13, a training data creation unit 14, a noise information creation unit 15, a data output unit 16, and a control unit 17. Deep learning is a form of machine learning. Furthermore, the data creation device according to the present invention can be configured to create training data for machine learning that enables speech recognition, as well as training data for machine learning that enables object recognition. However, as an example, the data creation device 1 that creates training data for machine learning that enables object recognition will be described below.

[0014] The data input unit 11 accepts input of real data obtained by photographing an object (a form of "object" according to the present invention) and virtual data modeled from the object. Real data is data obtained by photographing a real object and represents the appearance of the object. This real data can also be viewed as data obtained from a visual object obtained from physical space. Virtual data is data representing the appearance of a model obtained by modeling a real object or a virtual object. This modeling can be performed using, for example, CAD software.

[0015] Note that the real data and the virtual data can be either two-dimensional data or three-dimensional data representing an object. In the following description, as an example, it is assumed that the real data and the virtual data are two-dimensional data representing an object, and the data is data of a color image representing the appearance of the object (data including information of a plurality of pixels to which luminance of RGB (red, green, blue) is assigned). The real data, which is two-dimensional data, can be obtained, for example, by photographing an object with a digital camera or the like. The virtual data, which is two-dimensional data, may be, for example, image data of a 2D model of an object modeled by 2D CAD software, or image data of the appearance of a 3D model of an object modeled by 3D CAD software viewed from one direction.

[0016] The noise information acquisition unit 12 acquires noise information, which is information specific to the real data that is not included in the virtual data. That is, even for the real data and the virtual data of the same object, there is information that is included in the real data but not included in the virtual data, and the noise information acquisition unit 12 acquires this information as noise information. As an example of the content of the noise information, there are halation, occlusion, differences in labels attached to the object, and differences in the color of the object. The operation for acquiring the noise information (noise information acquisition operation) will be described in detail again.

[0017] The noise information holding unit 13 holds the noise information acquired by the noise information acquisition unit 12 and the noise information newly created by the noise information creation unit 15. As will be described later, the noise information held by the noise information holding unit 13 can be used for creating teacher data.

[0018] The teacher data creation unit 14 creates teacher data using the noise information held by the noise information holding unit 13. The operation for creating the teacher data (teacher data creation operation) will be described in detail again.

[0019] The noise information creation unit 15 synthesizes all or part of the multiple noise information held by the noise information storage unit 13 to create new noise information. This newly created noise information is stored in the noise information storage unit 13, just like other noise information. The operation for creating new noise information (noise information creation operation) will be explained in detail later.

[0020] The data output unit 16 outputs the training data created by the training data creation unit 14. An example of a destination for the training data output is a learning system that uses the training data to perform machine learning. In this case, the learning system can use the training data created by the data creation device 1 to perform machine learning for image recognition for object recognition.

[0021] The control unit 17 appropriately controls each part of the data creation device 1 to ensure that the data creation device 1 operates normally. The operations performed by the data creation device 1 include noise information acquisition operations for acquiring noise information, training data creation operations for creating training data, and noise information creation operations for creating new noise information. These operations will be described below.

[0022] 2. Operation of the data creation device, etc. (1) Noise information acquisition operation First, the noise information acquisition operation will be explained with reference to the flowchart shown in Figure 2. When the noise information acquisition operation is started, the data input unit 11 accepts input of real data for noise information acquisition (real data DA), which is real data obtained by photographing an arbitrary object, and virtual data for noise information acquisition (virtual data DB), which is virtual data obtained by modeling the same object (step S1).

[0023] Then, when the real data DA and virtual data DB are input, the noise information acquisition unit 12 acquires noise information NX that is included in the real data DA but not in the virtual data DB (step S2). More specifically, the noise information acquisition unit 12 calculates the difference (DA-DB) between the real data DA and the virtual data DB and acquires this difference as noise information NX.

[0024] Furthermore, it is preferable that the noise information acquisition unit 12 corrects the real data DA or virtual data DB to eliminate any slight discrepancies in the shape, size, or position of the object's outline between the real data DA and the virtual data DB, and then calculates the difference between the real data DA and the virtual data DB. However, various methods can be used to acquire noise information NX, as long as they can acquire information as noise information NX that is included in the real data but not in the virtual data, even for the same object. For example, artificial intelligence (AI) can be used to extract elements from the real data DA that are not present in the virtual data DB, and these can be used as noise information NX.

[0025] The noise information NX acquired by the noise information acquisition unit 12 is stored in the noise information storage unit 13 (step S3). In this way, the data creation device 1 acquires new noise information NX each time it performs a noise information acquisition operation and stores it in the noise information storage unit 13, making it possible to hold a large amount of noise information NX.

[0026] Figure 3 conceptually illustrates a specific example of how noise information acquisition is performed. In the example shown in Figure 3, noise information NX-1 is created based on real data DA-1 and virtual data DB-1. Noise information NX-1 is created as the difference between real data DA-1 and virtual data DB-1. In this embodiment, the "-n" (where n is a natural number) at the end of the code indicates that it is the nth specific example.

[0027] In the example shown in Figure 3, real data DA-1 and virtual data DB-1 represent images of the same object (the same smartphone), but real data DA-1 differs from virtual data DB-1 in that it mainly contains image elements (noise) consisting of halation N1. Therefore, when the noise information acquisition operation is performed and real data DA-1 and virtual data DB-1 are input to the noise information acquisition unit 12, the noise information acquisition unit 12 will acquire noise information NX-1, which mainly includes halation N1.

[0028] Figure 4 conceptually illustrates another specific example of how noise information acquisition is performed. In the example shown in Figure 4, noise information NX-2 is created based on real data DA-2 and virtual data DB-1. Noise information NX-2 is created as the difference between real data DA-2 and virtual data DB-1.

[0029] In the example shown in Figure 4, the real data DA-2 and virtual data DB-1 represent images of the same object (the same smartphone), but the real data DA-2 differs from the virtual data DB-1 in that it mainly contains image elements (noise) consisting of occlusion N2. Therefore, when the noise information acquisition operation is performed and the real data DA-2 and virtual data DB-1 are input to the noise information acquisition unit 12, the noise information acquisition unit 12 will acquire noise information NX-2, which mainly includes occlusion N2.

[0030] As shown in the examples in Figures 3 and 4, the data creation device 1 can acquire new noise information NX each time it performs a noise information acquisition operation. Therefore, by repeatedly performing the noise information acquisition operation while changing at least the real data DA, it is possible to create new noise information NX each time and store it in the noise information storage unit 13. It is preferable that the more noise information NX the noise information storage unit 13 holds, the more training data can be created by the training data creation operation described later.

[0031] (2) Training data creation process Next, the training data creation operation will be explained with reference to the flowchart shown in Figure 5. When the training data creation operation is started, the data input unit 11 accepts the input of virtual data DX of the recognition target, which is virtual data modeled after the recognition target (step S11). This recognition target is an object that will be used for image recognition. For example, when creating training data for machine learning that enables image recognition of a predetermined wallet A, wallet A should be used as the recognition target.

[0032] When the virtual data DX to be recognized is input, the training data creation unit 14 uses the virtual data DX to be recognized and the noise information NX held by the noise information holding unit 13 to create training data Z for the recognition target (step S12). The created training data Z is output externally from the data output unit 16 (step S13). The output training data Z can be used for machine learning for image recognition of the recognition target.

[0033] Furthermore, the noise information NX used to create the training data Z for the object to be recognized may be noise information NX derived from real data of an object different from the object to be recognized. For example, if the object to be recognized is a "wallet," the noise information NX used to create the training data Z for the object to be recognized is not limited to noise information NX derived from real data of a "wallet," but can also be noise information NX derived from real data of various objects such as a "smartphone." Here, an example of a specific method for creating the training data Z for the object to be recognized (step S12) will be explained below, using the methods of the first to third patterns as examples.

[0034] (2-1) Method of Pattern 1 One of the first patterns of methods is to create training data Z for the target of recognition by combining (combining) noise information NX with the virtual data DX to be recognized. Figure 6 conceptually shows a concrete example of how training data Z for the target of recognition is created using this first pattern of method.

[0035] In the example shown in Figure 6, the training data Z-1 for recognition is created by combining the virtual data DX-1 to be recognized with noise information NX-1. Furthermore, the training data Z-2 for recognition is created by combining the virtual data DX-1 to be recognized with noise information NX-2. According to the first pattern method described above, it is possible to create n training data Z using n noise information NX.

[0036] (2-2) Method of the second pattern Another approach involves generating synthesized virtual data DXa by combining noise information NX with virtual data DXa to be recognized, and then creating training data Z for the target of recognition by applying transformation processing to this synthesized virtual data DXa. Specific transformation processing applied to the synthesized virtual data DXa includes image scaling, rotation, translation, horizontal or vertical inversion, shear transformation, and color transformation.

[0037] Figure 7 conceptually illustrates a specific example of how the training data Z for recognition is created using the second pattern of method. In the example shown in Figure 7, synthesized virtual recognition data DXa-1 (created in the same way as training data Z-1 shown in Figure 6, so the illustration of the creation process is omitted) is generated by combining noise information NX-1 with the virtual recognition data DX-1. Training data Z-3 is created by applying an image reduction transformation process to this virtual recognition data DXa-1, and training data Z-4 is created by applying an image rotation transformation process.

[0038] Furthermore, in the example shown in Figure 7, a synthesized virtual data DXa-2 (created in the same way as the training data Z-2 shown in Figure 6, so the illustration of the creation process is omitted) is generated by combining noise information NX-2 with the virtual data DX-1 to be recognized. Training data Z-5 is created by applying an image reduction transformation process to this virtual data DXa-2, and training data Z-6 is created by applying an image rotation transformation process. According to the second pattern method described above, for example, it is possible to generate n synthesized virtual data DXa using n pieces of noise information NX, and then create n × m pieces of training data Z by applying m different transformation processes to each virtual data DXa.

[0039] (2-3) Third Pattern of Method A third pattern of method involves creating training data Z for recognition by applying a transformation process to the virtual data DX to be recognized to generate transformed virtual data DXb, and then combining noise information NX with the transformed virtual data DXb. Specific transformation processes applied to the virtual data DX include image scaling, rotation, translation, horizontal or vertical inversion, shear transformation, and color transformation.

[0040] Figure 8 conceptually illustrates a specific example of how the training data Z for recognition is created using the third pattern of method. In the example shown in Figure 8, training data Z-7 is created by combining noise information NX-1 with virtual recognition data DXb-1, which is obtained by applying an image reduction transformation to virtual recognition data DX-1. Furthermore, training data Z-8 is created by combining noise information NX-1 with virtual recognition data DXb-2, which is obtained by applying an image rotation transformation to virtual recognition data DX-1.

[0041] Furthermore, according to the third pattern method described above, for example, by applying n different transformation processes to the virtual data DX to be recognized to generate n transformed virtual data DXb, and then synthesizing m pieces of noise information NX with each virtual data DXb, it becomes possible to create n × m pieces of training data Z.

[0042] The training data creation unit 14 can create a large amount of training data Z for recognition by executing some or all of the methods described in the first to third patterns above. The method for creating training data Z adopted by the training data creation unit 14 may be specified by the user, and methods other than the first to third patterns described above may be adopted without departing from the spirit of the present invention. Furthermore, the method for creating training data Z adopted by the training data creation unit 14 may be combined with various publicly known methods (for example, the RandAugment method).

[0043] (3) Noise information creation operation Next, the noise information creation operation will be explained with reference to the flowchart shown in Figure 9. When the noise information creation operation is started, multiple noise information NX to be used to create new noise information are identified from the multiple noise information NX already held in the noise information holding unit 13 (step S21). This identification may be performed automatically by the data creation device 1, or it may be performed based on user specifications.

[0044] When multiple noise information NXs are identified, the noise information creation unit 15 uses these noise information NXs to create new noise information NXs (step S22). More specifically, the noise information creation unit 15 creates new noise information NXs by combining the identified noise information NXs. However, a different method may be used to create the new noise information NXs. The noise information NXs created by the noise information creation unit 15 are stored in the noise information storage unit 13 (step S23).

[0045] Figure 10 conceptually illustrates a specific example of how new noise information NX is created during the noise information generation process. In the example shown in Figure 10, new noise information NX-3, which includes both halation N1 and occlusion N2, is created by combining noise information NX-1, which mainly includes halation N1, with noise information NX-2, which mainly includes occlusion N2.

[0046] In the example described above, two noise information NX are combined to create new noise information NX, but it is also possible to combine three or more noise information NX to create new noise information NX. In this way, the noise information creation operation makes it possible to create new noise information NX based on the noise information NX held by the noise information holding unit 13, thereby increasing the amount of noise information NX held by the noise information holding unit 13. This makes it possible to create more training data by the training data creation operation described above. In addition, the noise information creation unit 15 may perform various transformation operations (for example, normalization, vertical inversion, horizontal inversion, rotation, or resizing (image quality change)) on the noise information NX already held by the noise information holding unit 13 (or the noise information NX created by the noise information creation operation described above) as other operations to increase the amount of noise information NX.

[0047] 3. Others As described above, the data creation device 1 is a data creation device for creating training data for machine learning, and comprises a data input unit 11 that accepts input of real data obtained by photographing an object and virtual data obtained by modeling an object, a noise information acquisition unit 12 that acquires noise information NX for the same object that is included in the real data DA but not in the virtual data DB, and a training data creation unit 14 that synthesizes the noise information NX with the virtual data DX of the recognition target obtained by modeling the recognition target to create training data Z of the recognition target.

[0048] Therefore, the data creation device 1 makes it possible to easily create a large amount of training data that enables proper object recognition. In other words, when real data of the object to be recognized (including data that has undergone known transformation processing) is used as training data, collecting a large amount of real data takes a lot of time and effort, so it is not easy to create a large amount of training data by simply using real data as training data. However, in this embodiment, training data is created based on virtual data of the object to be recognized, making it easy to create a large amount of training data.

[0049] Virtual data is easy to deform using CAD or simulation software used for modeling, making it suitable for creating large amounts of training data. Furthermore, even when creating training data for a recognition target that is still in the design stage, virtual data can be used to create training data without creating prototypes or other physical objects of the recognition target.

[0050] However, generally, object recognition is performed by photographing the object in real space. Therefore, there is a gap between the virtual data obtained by modeling the object and the information obtained by photographing the object. If machine learning is performed using only the virtual data of the object as training data, the problem may arise in that object recognition is not performed properly. In this embodiment, however, by utilizing not only the virtual data DX of the object to be recognized but also noise information NX derived from real data, this gap is minimized as much as possible, and training data Z that enables proper object recognition is created.

[0051] The specific form of the data creation device 1 is not particularly limited; for example, it may be a device dedicated to creating training data, or it may be a computer (such as a personal computer or workstation) equipped with various functions. In the latter case, a virtual data DB for noise information acquisition or virtual data DX for recognition targets may be formed by CAD software installed on the computer, and this may be input to the data input unit 11. Furthermore, if the computer has a function for performing machine learning for object recognition, the computer may perform machine learning using the training data Z output from the data output unit 16. In this case, the computer can also be viewed as a learning system that performs machine learning for object recognition using the training data Z created by the data creation device 1. The learning system that performs machine learning for object recognition using the training data Z may be configured on a computer separate from the data creation device 1.

[0052] As mentioned earlier, the real data and virtual data handled by the data creation device 1 may also be 3D data representing objects. In this case as well, it is possible to create training data Z in the same manner as when the real data and virtual data are 2D data.

[0053] Real data and virtual data, when they are 3D data, represent the three-dimensional shape of an object and can represent the area occupied by the object in 3D space (3D coordinates). Real data, which is 3D data, can be obtained, for example, by photographing an object with a 3D camera. As virtual data, which is 3D data, data from a 3D model created with 3D CAD software can be used, for example.

[0054] When real data and virtual data are 3D data, the noise information acquired by the noise information acquisition unit 12 may include, for example, differences in texture, bending, surface irregularities, and foreign matter such as dust and dirt. Examples of deformation processing applied to virtual data include various processes that deform the 3D shape entirely or partially.

[0055] As mentioned above, the data creation device according to the present invention can also be configured to create training data for machine learning that enables speech recognition (mechanical recognition where the recognition target is sound). In this case as well, the data creation device can create training data by performing operations similar in nature to those used when creating training data for machine learning that enables object recognition. This data creation device makes it possible to easily create a large amount of training data that enables speech recognition to be properly performed. In the data creation device that creates training data for machine learning that enables speech recognition, the data input unit should accept input of real data obtained by recording real sound (a form of "target" according to the present invention) (which can also be seen as data obtained from auditory targets obtained from physical space) and virtual data obtained by sound modeling (for example, sound data that can be created using sound creation software). In this case, both the real data and virtual data are speech data representing sound. The noise information acquisition unit should acquire, for example, the difference between the real data and virtual data for the same sound as noise information. In this case, the content of the noise information may include white noise or background noise. Examples of transformation processes applied to virtual data include various processes that alter the content of sound, either entirely or partially.

[0056] Furthermore, the data creation device according to the present invention may be configured to create training data for machine learning that enables recognition related to taste, smell, or touch (mechanical recognition where the recognition target is taste, smell, or texture). In this case, real data may be adopted as real data of objects related to taste, smell, or touch (taste, smell, or texture) obtained from physical space, and virtual data may be adopted as virtual data obtained by modeling objects related to taste, smell, or touch. The noise information acquisition unit may, for example, acquire the difference between real data and virtual data for the same taste, smell, or texture as noise information. In relation to the present invention, real data is data of objects obtained from physical space, virtual data is data obtained by modeling objects from virtual space, and recognition target virtual data is data obtained by modeling recognition targets from virtual space.

[0057] Although embodiments of the present invention have been described above, the configuration of the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the invention. The technical scope of the present invention is indicated not by the above description of embodiments, but by the claims, and should be understood to include all modifications that fall within the meaning and scope equivalent to the claims. [Industrial applicability]

[0058] This invention can be used in a data creation device for generating training data for machine learning. [Explanation of Symbols]

[0059] 1. Data creation device 11. Data Input Section 12. Noise information acquisition unit 13. Noise Information Storage Unit 14. Training Data Creation Department 15. Noise Information Creation Section 16. Data Output Section 17 Control Unit Real data for acquiring DA noise information Virtual data for acquiring DB noise information DX Recognition Target Virtual Data DXa synthesized virtual data to be recognized DXb-deformed virtual data to be recognized NX Noise Information

Claims

1. A data creation device for creating training data for machine learning, A data input unit that accepts real data of an object obtained from physical space and virtual data obtained from modeling an object in virtual space, A noise information acquisition unit that acquires the difference between the real data and the virtual data for the same subject as noise information that is included in the real data but not in the virtual data, A noise information holding unit that holds multiple pieces of the aforementioned noise information, A noise information creation unit synthesizes multiple noise pieces identified from the multiple noise pieces held by the noise information holding unit to create new noise pieces that include the contents of each of the identified noise pieces. The system includes a training data creation unit that synthesizes the noise information with virtual data of a recognition target obtained by modeling the recognition target from a virtual space to create the training data of the recognition target, A data creation device characterized in that the noise information held by the noise information holding unit, which is available for creating training data, is augmented when the new noise information is stored in the noise information holding unit.

2. The noise information acquisition unit, The data creation apparatus according to claim 1, characterized in that it corrects the real data or the virtual data so as to eliminate any discrepancies in the shape, size, or position of the contour of the object between the real data and the virtual data for the same object, and then calculates the difference.

Images