Information processing device and program

By using synthetic images generated from multiple real-world modalities, the accuracy of student networks is improved, addressing the issue of training data discrepancies and enhancing estimation performance.

JP7586172B2Active Publication Date: 2024-11-19SONY GROUP CORP
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022517550
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-04-30
Filing Date
2021-03-18
Publication Date
2024-11-19
Estimated Expiration
2041-03-18

AI Technical Summary

Technical Problem

Existing techniques for generating a student network from a teacher network without available training data often result in decreased accuracy due to discrepancies between training and real-world environments, necessitating improved quality of synthetic learning data.

Method used

Generating synthetic images using real environment images from multiple modalities and incorporating these images into the learning process to create a student network that can accurately estimate object classes in diverse conditions.

Benefits of technology

Enhances the accuracy of the student network's estimation capabilities by aligning the synthetic images with real-world features across various modalities, improving performance in real-world scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007586172000002
    Figure 0007586172000002
  • Figure 0007586172000003
    Figure 0007586172000003
  • Figure 0007586172000004
    Figure 0007586172000004
Patent Text Reader

Abstract

[Problem] To further improve the accuracy of an estimation using a student network. [Solution] Provided is an information processing device comprising an estimation unit that estimates an object class pertaining to an object included in an input image, by using a student network generated on the basis of a teacher network generated by machine learning using, as learning data, images saved in a large-scale image database. The student network is generated by machine learning using, as learning data, synthetic images obtained using the teacher network and actual environment images acquired by a plurality of different modalities for an actual environment where execution of estimation by the estimation unit is anticipated.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an information processing device and a program. [Background technology]

[0002] In recent years, neural networks, which are mathematical models that mimic the mechanism of the nervous system, have been attracting attention. Many techniques have also been proposed to improve the efficiency of learning using neural networks. For example, Non-Patent Document 1 discloses a technique for generating a Student Network from a Teacher Network even when the learning data used for learning the Teacher Network is unavailable. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Kartikeya Bhardwaj and 2 others, "Dream Distillation: A Data-Independent Model Compression Framework", May 17, 2019, [Online], [Retrieved April 1, 2020], Internet<https: / / arxiv.org / pdf / 1905.07072.pdf> Summary of the Invention [Problem to be solved by the invention]

[0004] When images generated using a teacher network are used to train a student network, as in the technology described in Non-Patent Document 1, it is important to improve the quality of the images in order to improve the accuracy of estimation by the student network. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided an information processing device including an estimation unit that estimates an object class associated with an object included in an input image using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as training data, wherein the student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation by the estimation unit is expected to be performed as training data.

[0006] According to another aspect of the present disclosure, there is provided an information processing device including a learning unit that generates a student network based on a teacher network generated by machine learning using images stored in a large-scale image database as training data, wherein the learning unit generates the student network by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation using the student network is expected to be performed, as training data.

[0007] According to another aspect of the present disclosure, there is provided a program for causing a computer to function as an information processing device, comprising: an estimation unit that estimates an object class associated with an object included in an input image using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as training data, wherein the student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation by the estimation unit is expected to be performed as training data. [Brief description of the drawings]

[0008] [Figure 1] FIG. 13 is a diagram for explaining object class estimation using a student network. [Diagram 2] FIG. 13 is a diagram for explaining the generation of a synthetic image using real environment images acquired by multiple modalities according to one embodiment of the present disclosure. [Diagram 3] FIG. 2 is a diagram for explaining an overview of four phases according to the embodiment. [Figure 4] 2 is a block diagram showing an example of the configuration of a first learning device 10 according to the embodiment. FIG. [Diagram 5] FIG. 2 is a block diagram showing an example of the configuration of an image generating device 20 according to the embodiment. [Figure 6] 13 is a flowchart showing the flow of synthetic image generation using the image generation unit 210 according to the embodiment. [Figure 7] 2 is a block diagram showing an example of the configuration of a second learning device 30 according to the embodiment. FIG. [Figure 8] 13 is a diagram for explaining synthetic images with high similarity between different modalities according to the embodiment. FIG. [Figure 9] 13 is a flowchart showing a flow of determining whether or not to accept a synthetic image based on similarity between modalities and learning according to the embodiment; [Figure 10] 13 is a diagram for explaining synthetic images with high similarity within the same modality according to the embodiment. FIG. [Figure 11] 13 is a flowchart showing a flow of determining whether or not to accept a synthetic image based on similarity within a modality and learning according to the embodiment; [Figure 12] FIG. 13 is a diagram showing an example of a fusion image according to the embodiment. [Figure 13] 13 is a flowchart showing the flow of generation and learning of a fusion image within the same modality according to the embodiment. [Figure 14]FIG. 13 is a diagram for explaining a fusion image focusing on multiple different modalities according to the embodiment. [Figure 15] 13 is a flowchart showing the flow of generating and learning a fusion image between different modalities according to the embodiment. [Figure 16] 13A to 13C are diagrams illustrating an example of a process for similarly processing feature amount distributions related to each modality according to the embodiment. [Figure 17] FIG. 2 is a block diagram showing an example of the configuration of an estimation device 40 according to the embodiment. [Figure 18] 6 is a flowchart showing a flow of processing by the estimation device 40 according to the embodiment. [Figure 19] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device 90 according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] The explanation will be given in the following order. 1. Embodiment 1.1. Background 1.2. Creating a teacher network 1.3.Generating synthetic images 1.4. Creating a Student Network 1.5. Estimation using Student Networks 2. Hardware configuration example 3. Summary

[0011] <1. Embodiment> <<1.1.Background>> First, an overview of an embodiment of the present disclosure will be described. As described above, in recent years, many techniques have been proposed for improving the efficiency of learning using neural networks. For example, one such technique is known as knowledge distillation.

[0012] Knowledge distillation is a technique for generating a new network (student network) according to the required specifications from a highly accurate trained network (teacher network).

[0013] Knowledge distillation is used, for example, when it is desired to generate a student network that is more suited to a real environment from a general-purpose teacher network that has been trained using images provided from a large-scale image database.

[0014] However, in order to perform knowledge distillation, the learning data used for training the teacher network is generally required. Therefore, if the learning data used for training the teacher network is not available or if the amount of available learning data is insufficient, it may be difficult to generate a student network or the accuracy of the generated student network may be reduced.

[0015] On the other hand, the above-mentioned Non-Patent Document 1 discloses a technique for generating a student network without using the learning data used in learning the teacher network.

[0016] The technology disclosed in Non-Patent Document 1 generates a synthetic image by inputting a real-environment image acquired in the real environment into a teacher network, and performs machine learning using the synthetic image as training data to generate a student network.

[0017] According to this technology, even in a situation where the learning data used to generate the teacher network is unavailable, it is possible to generate a student network based on the teacher network. Details of synthetic images will be described separately later.

[0018] Here, in generating a student network using synthetic images as described above, in order to further increase the accuracy of the generated student network, it is important to improve the quality of the synthetic images used as learning data.

[0019] Here, we consider the case of generating a student network that estimates the object class of an object contained in an input image.

[0020] Fig. 1 is a diagram for explaining the estimation of an object class using a student network. Fig. 1 illustrates a case where an estimation device (not shown) that performs estimation using a student network generated using a synthetic image is mounted on a moving object V such as an automobile.

[0021] The estimation device mounted on the moving body V estimates, for example, the object class of an object O included in an input photographed image in real time and outputs the estimation result. The object O may be, for example, a wild animal such as a rabbit. According to the estimation described above, it becomes possible to detect that a rabbit has jumped out in the traveling direction of the moving body V and to cause the moving body V to take an evasive action or the like.

[0022] However, the accuracy of estimating the object class using the student network is greatly affected by the data used during learning. For example, if there is a large discrepancy between the shooting environment (location, illuminance, distance, noise, object type, etc.) of the images used as learning data and the shooting environment of the images acquired in the real environment where estimation is actually performed, the student network may erroneously estimate the object class of object O.

[0023] For this reason, the learning data used in generating the student network must be close to images actually captured in the real environment. This also applies when using the synthetic images described above as learning data.

[0024] The technical idea of ​​the present disclosure has been conceived with a focus on the above points, and makes it possible to further improve the accuracy of estimation using a student network.

[0025] To this end, in one embodiment of the present disclosure, a synthetic image may be generated using real environment images acquired in a real environment by multiple modalities, and a student network may be generated using the synthetic image.

[0026] 2 is a diagram for explaining generation of a synthetic image using real environment images acquired by multiple modalities according to an embodiment of the present disclosure. In FIG. 2, an example is shown in which synthetic images S1 to S3 corresponding to different modalities are generated using teacher networks T1 to T3 corresponding to three different modalities.

[0027] Here, the modality according to an embodiment of the present disclosure refers to a means for capturing an image. This expression is also widely used in the medical field, for example, to express the same meaning. Examples of the modality according to this embodiment include an RGB camera, a ToF camera, a LiDAR, and a polarization camera.

[0028] For example, when Modality 1 shown in FIG. 2 is an RGB camera, in the teacher network T1, a synthetic image S1 that imitates an RGB image is updated and acquired using pseudo-features generated by analyzing the distribution of features obtained from multiple RGB images captured by an RGB camera in a real environment so that the features obtained when a synthetic image is input to the teacher network T1 are close to the pseudo-features.

[0029] Furthermore, for example, when Modality 2 shown in FIG. 2 is a ToF camera, in the teacher network T2, a synthetic image S2 that imitates a ToF image is updated and acquired using pseudo-features generated by analyzing the distribution of features obtained from multiple ToF images captured by a ToF camera in a real environment, so that the features obtained when a certain synthetic image is input to the teacher network T2 are close to the pseudo-features.

[0030] Furthermore, for example, when Modality 3 shown in FIG. 2 is a polarization camera, in the teacher network T3, a synthetic image S3 that imitates a polarized image is updated and acquired using pseudo-features generated by analyzing the distribution of features obtained from multiple polarization images captured by a polarization camera in a real environment, so that the features obtained when a certain synthetic image is input to the teacher network T3 are close to the pseudo-features.

[0031] In the synthetic images S1 to S3 shown in FIG. 2, differences in modality are expressed by the type of texture or the presence or absence of texture.

[0032] In this manner, a synthetic image according to one embodiment of the present disclosure may be generated using real environment images acquired through different modalities and a teacher network.

[0033] The synthetic image in this embodiment can be generated, for example, by inputting a real environment image acquired by a corresponding modality to each of multiple teacher networks T1 to T3 corresponding to a single modality, as shown in Figure 2.

[0034] Moreover, one of the features of the student network according to an embodiment of the present disclosure is that it is generated by using the synthetic images generated as described above as learning data.

[0035] According to the above features, the student network can be trained broadly and effectively to learn the characteristics of object O that may appear in images acquired by each modality (e.g., information on shape, color, distance, deflection, etc.), thereby making it possible to improve the estimation accuracy of the student network.

[0036] The realization of estimation using a student network according to an embodiment of the present disclosure is roughly divided into four phases. Fig. 3 is a diagram for explaining an overview of the four phases according to an embodiment of the present disclosure.

[0037] Phase F1 according to one embodiment of the present disclosure is a phase for generating a teacher network.

[0038] Phase F2 according to one embodiment of the present disclosure is a phase for generating a synthetic image using the teacher network generated in Phase F1.

[0039] Phase F3 according to one embodiment of the present disclosure is a phase for generating a teacher network using the synthetic image generated in phase F2.

[0040] Phase F4 according to an embodiment of the present disclosure is a phase in which estimation is performed using the student network generated in phase F3.

[0041] The processing in each of the above phases will now be described in detail.

[0042] <<1.2. Creating a teacher network>> First, a method for generating a teacher network in Phase F1 will be described. In Phase F1 according to this embodiment, a teacher network is generated using a first learning device 10.

[0043] Fig. 4 is a block diagram showing a configuration example of the first learning device 10 according to this embodiment. As shown in Fig. 4, the first learning device 10 according to this embodiment includes at least a first learning unit 110. Note that the first learning device 10 may further include, for example, an operation unit that accepts operations by a user and a display unit that displays various information.

[0044] (First learning unit 110) The first learning unit 110 according to this embodiment performs machine learning using images stored in the large-scale DB 50 as learning data, and generates a teacher network.

[0045] A detailed description of the flow is omitted for the generation of the teacher network by the first learning unit 110, since a commonly used method may be adopted. The large-scale DB 50 used for generating the teacher network may be, for example, various databases that are published on the Internet and are widely used in research and development in the field of machine learning. By generating the teacher network using data with high estimation accuracy, it is possible to improve the accuracy of estimation by the student network generated later.

[0046] The first learning device 10 may be set in an environment different from that of the image generating device 20 and the second learning device 30 described below.

[0047] <<1.3. Creating synthetic images>> Next, a method for generating a synthetic image in Phase F2 will be described. In Phase F2 according to this embodiment, a synthetic image is generated using the image generating device 20.

[0048] Fig. 5 is a block diagram showing an example of the configuration of the image generating device 20 according to this embodiment. As shown in Fig. 5, the image generating device 20 according to this embodiment includes at least an image generating unit 210. Note that the first learning device 10 may further include, for example, an operation unit that accepts operations by a user and a display unit that displays various information.

[0049] Moreover, the image generating device 20 according to this embodiment is installed at a main development site, which includes a cloud environment.

[0050] (Image generation unit 210) The image generation unit 210 in this embodiment generates a synthetic image using the teacher network generated in phase F1 and real environment images acquired by multiple different modalities in the real environment where estimation is expected to be performed by the student network generated in phase 3.

[0051] The image generating unit 210 according to this embodiment generates a synthetic image using, for example, a real environment image stored in the real environment DB 60, and stores the generated synthetic image in the generated image DB .

[0052] The synthetic image according to this embodiment may be generated by adding noise to the features obtained by inputting a real-world image into a teacher network.

[0053] The generation of a synthetic image by the image generating unit 210 according to this embodiment will be described in detail below. Fig. 6 is a flowchart showing the flow of synthetic image generation using the image generating unit 210 according to this embodiment.

[0054] In phase F2 according to this embodiment, first, the modality and object class related to the synthetic image to be generated are designated (S202). This designation may be performed by a user (developer, etc.).

[0055] Next, the image generator 210 acquires a corresponding real environment image from the real environment DB 60 based on the modality and object class specified in step S202 (S204).

[0056] Next, the image generator 210 inputs the real environment image acquired in step S204 to the teacher network to acquire features (S206). The features may be features after average pooling in the teacher network.

[0057] Next, the image generating unit 210 projects the feature amount acquired in step S206 into a certain space using t-SNE or the like (S208).

[0058] The image generating unit 210 repeats the processes in steps S204 to S208 until a sufficient number of data items are collected.

[0059] Here, when a sufficient amount of data has been collected, the image generating unit 210 obtains a prescribed number of cluster centers and performs K-means (S210).

[0060] Next, the image generating unit 210 obtains principal components for the cluster data obtained in step S210 (S212). Note that the projection onto the space using the above t-SNE or the like does not necessarily have to be performed in step S208.

[0061] Next, the image generating unit 210 generates new features by adding noise in the direction of the principal component found in step S212 (S214).

[0062] In other words, the synthetic image according to this embodiment is generated by adding noise in the principal component direction in the feature distribution obtained by inputting the real environment image into the teacher network.

[0063] Next, the image generating unit 210 generates a synthetic image such that the difference between the feature amount acquired in step S206 and the feature amount generated in step S214 is small (S216).

[0064] In other words, the synthetic image of this embodiment is generated so as to reduce the difference between the features after average pooling obtained by inputting a real environment image into a teacher network and the features obtained by adding noise in the principal component direction to the features.

[0065] The above process can be expressed by the following equation (1).

[0066]

number

[0067] In addition, X in the above formula (1) i The synthetic image to be generated is g(X i ) is the feature value after average pooling in the teacher network when X is input, t i respectively represent feature amounts generated by adding noise in the principal component direction.

[0068] The image generating unit 210 according to this embodiment repeatedly executes the processes in steps S214 and S216 until a predetermined number of synthetic images are generated.

[0069] Furthermore, when generating a synthetic image relating to another modality or object class, the image generating unit 210 may return to step S202 and repeat the following processing.

[0070] The flow of synthetic image generation according to this embodiment has been described above. For more detailed information on the method, please refer to the above-mentioned Non-Patent Document 1.

[0071] <<1.4. Creating a Student Network>> Next, a method for generating a student network in phase F3 will be described. In phase F3 according to this embodiment, a student network is generated using a second learning device 30. The second learning device 30 according to this embodiment is an example of an information processing device that generates a student network by machine learning using the synthetic images generated in phase F2 as learning data.

[0072] Fig. 7 is a block diagram showing a configuration example of the second learning device 30 according to this embodiment. As shown in Fig. 7, the second learning device 30 according to this embodiment includes at least a second learning unit 310. Note that the first learning device 10 may further include, for example, an operation unit that accepts operations by a user and a display unit that displays various information.

[0073] Moreover, the second learning device 30 according to this embodiment is installed at a main development site. The development site includes a cloud environment.

[0074] (Second learning unit 310) The second learning unit 310 according to this embodiment is an example of a learning unit that generates a student network based on a teacher network generated by machine learning using images stored in the large-scale DB 50 as learning data. Also, one of the features of the second learning unit 310 according to this embodiment is that it generates a student network by machine learning using synthetic images obtained using a teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation using the student network is expected to be performed as learning data.

[0075] For example, the second learning unit 310 according to the present embodiment acquires the synthetic image generated in phase F2 from the generated image DB 70, and generates a student network by using the synthetic image as learning data. The second learning unit 310 may also use the real environment image stored in the real environment DB as learning data.

[0076] In this case, the second learning unit 310 of this embodiment may use only synthetic images stored in the generated image DB70 that are predicted to further improve the accuracy of estimation by the student network and real environment images as learning data.

[0077] By selecting the synthetic image as described above, it is possible to more effectively improve the accuracy of estimation by the generated student network.

[0078] For example, the second learning unit 310 according to this embodiment may use, as learning data, synthetic images whose similarity between different modalities exceeds a threshold, among the generated synthetic images.

[0079] Fig. 8 is a diagram for explaining synthetic images with high similarity between different modalities according to this embodiment. Fig. 8 illustrates synthetic images S11-S13, S21-S23, and S31-S33 related to the object class "rabbit". Here, the synthetic images S11-S13, S21-S21, and S31-S33 relate to different modalities.

[0080] In the example shown in Fig. 8, the rabbit subject in synthetic images S22 and S32 is missing one ear, and has low similarity to synthetic image S12. In this way, synthetic images with low similarity between different modalities may be excluded from the training data.

[0081] It is expected that the accuracy of estimation by the generated Student Network will be effectively improved by determining whether or not to accept a synthetic image based on the similarity between different modalities as described above.

[0082] Furthermore, the above-described acceptance / rejection judgment is expected to be particularly effective when data of approximately the same shape can be acquired in all modalities.

[0083] The similarity may be calculated using a technique such as template matching, feature point matching, etc. Furthermore, deep learning for determining similar images may be used for the similarity calculation.

[0084] FIG. 9 is a flowchart showing the flow of determining whether or not to accept a synthetic image based on the similarity between modalities and learning according to this embodiment.

[0085] First, the second learning unit 310 calculates similarities between different modalities for the generated synthetic images (S302).

[0086] Next, the second learning unit 310 excludes synthetic images whose similarity calculated in step S302 is equal to or less than a threshold (S304).

[0087] Next, the second learning unit 310 performs learning using the remaining synthetic images and real environment images as learning data to generate a student network (S306).

[0088] As described above, the student network according to this embodiment may be generated by machine learning using synthetic images whose similarity between different modalities exceeds a threshold value, among the generated synthetic images.

[0089] On the other hand, the second learning unit 310 according to this embodiment may use, as learning data, synthetic images whose similarity within the same modality exceeds a threshold, among the generated synthetic images.

[0090] Fig. 10 is a diagram for explaining synthetic images with high similarity within the same modality according to this embodiment. Fig. 10 illustrates synthetic images S11-S13, S21-S23, and S31-S33 related to the object class "rabbit". Here, the synthetic images S11-S13, S21-S21, and S31-S33 relate to different modalities.

[0091] In the example shown in Fig. 10, the synthetic image S23 is missing one ear of the rabbit subject, and has a low similarity to the synthetic images S21 and S22 related to the same modality. Similarly, the synthetic image S31 is missing one ear of the rabbit subject, and has a low similarity to the synthetic images S32 and S33 related to the same modality. In this way, synthetic images with low similarity within the same modality may be excluded from the training data.

[0092] It is expected that the above-described method of determining whether or not to accept a synthetic image based on similarity within the same modality will effectively improve the accuracy of estimation by the generated Student network.

[0093] The above-mentioned acceptance / rejection determination is expected to be particularly effective when the shape change within a modality is small. In this case, the acceptance / rejection determination based on the similarity within the same modality is effective even when the feature distributions of the modalities are not similar, unlike the acceptance / rejection determination based on the similarity between different modalities.

[0094] FIG. 11 is a flowchart showing the flow of synthetic image adoption / rejection determination and learning based on similarity within a modality according to this embodiment.

[0095] First, the second learning unit 310 calculates the similarity of the generated synthetic images within the same modality (S312).

[0096] Next, the second learning unit 310 excludes synthetic images whose similarity calculated in step S312 is equal to or less than a threshold (S314).

[0097] Next, the second learning unit 310 performs learning using the remaining synthetic images and real environment images as learning data to generate a student network (S316).

[0098] As described above, the student network according to this embodiment may be generated by machine learning using synthetic images whose similarity within the same modality exceeds a threshold value, among the generated synthetic images.

[0099] In the above, a case has been described in which a synthetic image to be adopted as learning data is selected from a plurality of generated synthetic images. Conversely, the second learning unit 310 according to the present embodiment may generate a new synthetic image by fusing the plurality of generated synthetic images, and use the new synthetic image as learning data. Hereinafter, the new synthetic image is referred to as a fusion image.

[0100] That is, the student network according to this embodiment may be generated by machine learning using a fusion image obtained by fusing a plurality of generated synthetic images.

[0101] Fig. 12 is a diagram showing an example of a fusion image according to the present embodiment. Fig. 12 illustrates a fusion image S13 generated by fusing synthetic images S11 and S12 related to the same modality. Similarly, Fig. 12 illustrates a fusion image S23 generated by fusing synthetic images S21 and S22 related to the same modality. Similarly, Fig. 12 illustrates a fusion image S33 generated by fusing synthetic images S31 and S32 related to the same modality.

[0102] As illustrated in FIG. 12, the fusion image according to this embodiment may be generated by fusing a plurality of synthetic images related to the same object class among the generated synthetic images.

[0103] Furthermore, the fusion image according to this embodiment may be generated by fusing a plurality of synthetic images related to the same modality that have been generated and whose similarity exceeds a threshold value.

[0104] In the above case, processing such as alignment and rotation may be performed to increase the similarity.

[0105] Also, if the similarity is sufficiently high, a fusion image may be generated by fusing synthetic images related to different modalities.

[0106] By generating a fusion image as described above, it is possible to create data that cannot normally be expressed in feature space, which may increase the diversity of the training data.

[0107] FIG. 13 is a flowchart showing the flow of generating and learning a fusion image related to the same modality according to this embodiment.

[0108] First, the second learning unit 310 calculates the similarity of the generated synthetic images within the same modality (S322).

[0109] Next, the second learning unit 310 fuses the synthetic images whose similarities calculated in step S322 exceed a threshold value to generate a fusion image (S324).

[0110] Next, the second learning unit 310 performs learning using the synthetic image including the fusion image generated in step S324 and the real environment image as learning data to generate a student network (S326). Note that in this case, an image obtained by fusing multiple real environment images may be used as learning data.

[0111] Next, a fusion image focusing on a plurality of different modalities according to the present embodiment will be described. Fig. 14 is a diagram for explaining a fusion image focusing on a plurality of different modalities according to the present embodiment.

[0112] FIG. 14 illustrates synthetic images S11, S21, and S31, which are synthetic images relating to different modalities, and a fusion image S41 generated by concatenating the synthetic images S11, S21, and S31.

[0113] In this manner, a fusion image according to this embodiment may be generated by concatenating synthetic images of multiple different modalities.

[0114] More specifically, the fusion image according to this embodiment may be generated by concatenating, in the channel direction, a plurality of synthetic images among the generated synthetic images whose similarity between different modalities exceeds a threshold value.

[0115] In the above case, processing such as alignment and rotation may be performed to increase the similarity.

[0116] The generation of the above-mentioned fusion image is expected to be particularly effective when data of approximately the same shape can be obtained in all modalities. In addition, the above-mentioned fusion image is expected to have a high possibility of improving the accuracy of estimation by the generated Student Network, since information from all modalities can be utilized.

[0117] FIG. 15 is a flowchart showing the flow of generating and learning a fusion image between different modalities according to this embodiment.

[0118] First, the second learning unit 310 calculates similarities between different modalities for the generated synthetic images (S332).

[0119] Next, the second learning unit 310 generates a fusion image by concatenating, in the channel direction, the synthetic images whose similarity calculated in step S332 exceeds the threshold (S334).

[0120] Next, the second learning unit 310 performs learning using the synthetic image including the fusion image generated in step S334 and the real environment image as learning data to generate a student network (S336). In this case, an image obtained by concatenating multiple real environment images related to different modalities in the channel direction may be used as learning data.

[0121] The above describes a concrete example of generating synthetic images according to the present embodiment. By using the synthetic images described above as training data, it is possible to effectively improve the accuracy of estimation by the generated student network.

[0122] When performing processing based on the similarity of synthetic images between different modalities as exemplified above, it is desirable that the feature distributions of each modality are similar.

[0123] However, in reality, there is no guarantee that the feature distributions are consistent across modalities. Therefore, if no processing is performed, it is possible that similar images will not be obtained, or that even if the noise values ​​used to generate synthetic images are the same, they may represent different distances or directions in the feature space.

[0124] In view of the above, the synthetic image according to the present embodiment may be generated based on a process of similarizing the distribution of features related to each modality obtained by inputting real-world images to a teacher network. The process may be performed on real-world images related to multiple modalities acquired at the same time and from the same direction.

[0125] The above processing makes it possible to absorb differences in feature distribution between modalities and generate synthetic images that are more useful as learning data.

[0126] 16 is a diagram showing an example of a process for similarizing feature distributions related to each modality according to the present embodiment, in which real environment images related to a plurality of modalities are acquired at the same timing and from the same direction.

[0127] In addition, in the example shown in Figure 16, the first learning unit 110 generates an image E11 by connecting multiple real environment images acquired as described above in the channel direction, and generates a teacher network T4 by machine learning using the image E11 as learning data.

[0128] By using the teacher network T4 generated as described above, it is possible to generate synthetic images S41 relating to all modalities used to generate the image E11.

[0129] In addition, the process of making the feature distributions related to each modality similar may include a process of reducing the distance in feature space between a feature obtained by inputting a real environment image related to a certain modality into a teacher network and a feature obtained by inputting a real environment image related to a modality different from the modality into a teacher network.

[0130] The above process includes, for example, domain adaptation. s Let X be the feature quantity obtained from the real environment image captured by the RGB camera. t is a feature obtained from a real environment image captured by a ToF camera, the first learning unit 110 calculates X s Distribution of and X t Learning can be performed so that the distribution approaches that of

[0131] In addition, the process of making the feature distributions related to each modality similar may include a process of converting features obtained by inputting a real environment image related to a certain modality into a teacher network into features obtained by inputting a real environment image related to a modality different from the said modality into the teacher network.

[0132] The first learning unit 110 in this embodiment may, for example, use a machine learning technique or the like to obtain a transformation matrix that projects a feature relating to a certain modality onto a feature relating to another modality, and use the transformation matrix to change the feature.

[0133] For example, X s Let X be the feature quantity obtained from the real environment image captured by the RGB camera. t is a feature obtained from a real environment image captured by a ToF camera, the first learning unit 110 calculates X t =A Xs +B, the feature quantity can be changed. Although a linear regression problem is shown as an example here, the transformation matrix may be obtained by a nonlinear regression problem.

[0134] A specific example of the process for making the feature distributions related to each modality similar to each other according to the present embodiment has been described above. Note that the above is merely an example, and other means may be used to improve the similarity of the feature distributions related to each modality.

[0135] <<1.5. Estimation using Student Network>> Next, estimation using the student network in phase F4 will be described. In phase F4 according to the present embodiment, estimation is performed using the student network generated in phase F3. The estimation device 40 according to the present embodiment is an example of an information processing device that estimates an object class using a student network.

[0136] Fig. 17 is a block diagram showing a configuration example of the estimation device 40 according to this embodiment. As shown in Fig. 17, the estimation device 40 according to this embodiment includes at least an acquisition unit 410 and an estimation unit 420. Note that the estimation device 40 may further include, for example, an operation unit that accepts operations by a user and a display unit that displays various information. Furthermore, the estimation device 40 may be mounted on a moving body such as an automobile.

[0137] (Acquisition section 410) The acquisition unit 410 according to the present embodiment acquires an image in a real environment. More specifically, the acquisition unit 410 according to the present embodiment acquires an image by at least one modality among a plurality of modalities used to acquire a real environment image used to generate a synthetic image used in training a student network used by the estimation unit 420.

[0138] For example, if an RGB camera and a ToF camera are used to acquire the real environment image used to generate the synthetic image, the acquisition unit 410 acquires the image using at least one of the RGB camera or the ToF camera.

[0139] (Estimation section 420) The estimation unit 420 according to this embodiment estimates an object class for an object contained in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data.

[0140] For example, the estimation unit 420 according to the present embodiment may estimate an object class related to an object included in an image acquired by the acquisition unit 410. If there is an image acquired by at least one modality among the modalities used to acquire the real environment image used to generate the synthetic image, the estimation unit 420 according to the present embodiment can input the image to the student network and output an estimation result.

[0141] The flow of estimation by the estimation device 40 according to this embodiment will be described below. Fig. 18 is a flowchart showing the flow of processing by the estimation device 40 according to this embodiment.

[0142] As shown in FIG. 18, first, the acquisition unit 410 acquires an image in the same modality as that used to acquire the real environment image used to generate the synthetic image (S402).

[0143] Next, the estimation unit 420 inputs the image acquired in step S402 into the student network and performs object class estimation of the object included in the image (S406).

[0144] Next, the estimation unit 420 outputs the estimation result (S406). The estimation result may be displayed on a display unit or the like provided on the estimation device 40 or the moving object, or may be used to control each device (e.g., brake, steering, etc.) provided on the moving object, for example.

[0145] <2. Hardware configuration example> Next, a hardware configuration example common to the first learning device 10, the image generating device 20, the second learning device 30, and the estimation device 40 according to an embodiment of the present disclosure will be described. FIG. 19 is a block diagram showing a hardware configuration example of an information processing device 90 according to an embodiment of the present disclosure. The information processing device 90 may be a device having a hardware configuration equivalent to each of the above devices. As shown in FIG. 19, the information processing device 90 has, for example, a processor 871, a ROM 872, a RAM 873, a host bus 874, a bridge 875, an external bus 876, an interface 877, an input device 878, an output device 879, a storage 880, a drive 881, a connection port 882, and a communication device 883. Note that the hardware configuration shown here is an example, and some of the components may be omitted. In addition, the information processing device 90 may further include components other than the components shown here.

[0146] (Processor 871) The processor 871 functions, for example, as an arithmetic processing device or control device, and controls all or part of the operation of each component based on various programs recorded in the ROM 872, the RAM 873, the storage 880, or the removable recording medium 901.

[0147] (ROM 872, RAM 873) The ROM 872 is a means for storing the programs to be read into the processor 871, data to be used for calculations, etc. The RAM 873 temporarily or permanently stores, for example, the programs to be read into the processor 871, various parameters that change appropriately when the programs are executed, etc.

[0148] (host bus 874, bridge 875, external bus 876, interface 877) The processor 871, ROM 872, and RAM 873 are connected to one another via, for example, a host bus 874 capable of high-speed data transmission. On the other hand, the host bus 874 is connected to an external bus 876 having a relatively low data transmission speed via, for example, a bridge 875. In addition, the external bus 876 is connected to various components via an interface 877.

[0149] (Input Device 878) The input device 878 may be, for example, a mouse, a keyboard, a touch panel, a button, a switch, a lever, etc. Furthermore, a remote controller capable of transmitting a control signal using infrared rays or other radio waves may be used as the input device 878. The input device 878 may also include an audio input device such as a microphone.

[0150] (Output Device 879) The output device 879 is a device capable of visually or audibly notifying the user of acquired information, such as a display device such as a CRT (Cathode Ray Tube), LCD, or organic EL, an audio output device such as a speaker or a headphone, a printer, a mobile phone, a facsimile, etc. The output device 879 according to the present disclosure also includes various vibration devices capable of outputting tactile stimuli.

[0151] (Storage 880) The storage 880 is a device for storing various types of data. For example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device may be used as the storage 880.

[0152] (Drive 881) The drive 881 is a device that reads information recorded on a removable recording medium 901 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, or writes information to the removable recording medium 901.

[0153] (Removable recording media 901) The removable recording medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, various semiconductor storage media, etc. Of course, the removable recording medium 901 may be, for example, an IC card equipped with a non-contact type IC chip, an electronic device, etc.

[0154] (Connection port 882) The connection port 882 is a port for connecting an external device 902, such as a Universal Serial Bus (USB) port, an IEEE 1394 port, a Small Computer System Interface (SCSI), an RS-232C port, or an optical audio terminal.

[0155] (External connection device 902) The external connection device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.

[0156] (Communication Device 883) The communication device 883 is a communication device for connecting to a network, such as a communication card for wired or wireless LAN, Bluetooth (registered trademark), or WUSB (Wireless USB), a router for optical communications, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communications.

[0157] <3. Summary> As described above, the estimation device 40 according to an embodiment of the present disclosure includes an estimation unit 420 that estimates an object class associated with an object included in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as training data.

[0158] Moreover, the second learning device 30 according to an embodiment of the present disclosure includes a second learning unit 310 that generates a student network based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data. One of the features of the second learning unit 310 according to an embodiment of the present disclosure is that it generates a student network by machine learning using synthetic images obtained using a teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation using the student network is expected to be performed as learning data.

[0159] According to the above configuration, it is possible to further improve the accuracy of estimation using the student network.

[0160] Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

[0161] For example, the present technology may be applied to an image in which a sound source is visualized. In recent years, various technologies for generating an image from a sound source have been developed. For example, the image generating device 20 may generate a synthetic image using an image in which a sound source is visualized, and the second learning device 30 may generate a student network using the synthetic image. In this case, the estimation device 40 may estimate a class related to the sound source by using the student network generated as described above.

[0162] Furthermore, the steps of the processes described in this specification do not necessarily have to be processed in chronological order according to the order described in the flowchart or sequence diagram. For example, the steps of the processes of each device may be processed in an order different from the order described, or may be processed in parallel.

[0163] Furthermore, the series of processes performed by each device described in this specification may be realized using any of software, hardware, and a combination of software and hardware. The programs constituting the software are stored in advance, for example, in a recording medium (non-transitory medium) provided inside or outside each device. Then, each program is loaded into a RAM when executed by a computer, for example, and executed by various processors. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. Furthermore, the computer programs may be distributed, for example, via a network, without using a recording medium.

[0164] In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

[0165] Note that the following configurations also fall within the technical scope of the present disclosure. (1) an estimation unit that estimates an object class of an object included in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data; Equipped with The student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which the estimation by the estimation unit is expected to be performed, as learning data. Information processing device. (2) The synthetic image is generated based on adding noise to features obtained by inputting the real environment image to the teacher network. The information processing device according to (1). (3) The synthetic image is generated based on adding noise in the principal component direction in a feature distribution obtained by inputting the real environment image to the teacher network. The information processing device according to (2). (4) The synthetic image is generated so that the difference between a feature amount after average pooling obtained by inputting the real environment image to the teacher network and a feature amount obtained by adding noise to the feature amount in the principal component direction is small. The information processing device according to (3). (5) The student network is generated by machine learning using the synthetic images whose similarity between different modalities exceeds a threshold among the generated synthetic images. The information processing device according to any one of (1) to (4). (6) The student network is generated by machine learning using the synthetic images whose similarity within the same modality exceeds a threshold among the generated synthetic images. The information processing device according to any one of (1) to (5). (7) The student network is generated by machine learning using a fusion image obtained by fusing a plurality of the generated synthetic images. The information processing device according to any one of (1) to (6). (8) The fusion image is generated by fusing a plurality of the synthetic images whose similarity exceeds a threshold value among the generated synthetic images. The information processing device according to (7). (9) The fusion image is generated by fusing a plurality of the synthetic images related to the same object class among the generated synthetic images. The information processing device according to (7) or (8). (10) The fusion image is generated by concatenating the synthetic images of multiple different modalities. The information processing device according to any one of (7) to (9). (11) The fusion image is generated by concatenating, in a channel direction, a plurality of the synthetic images whose similarities between different modalities exceed a threshold value among the generated synthetic images. The information processing device according to (10). (12) The synthetic image is generated based on inputting the real environment image acquired by a corresponding modality to each of a plurality of teacher networks corresponding to a single modality. The information processing device according to any one of (1) to (11). (13) The synthetic image is generated based on a process of similarifying feature distributions related to each modality obtained by inputting the real environment image into the teacher network. The information processing device according to any one of (1) to (11). (14) The synthetic image is generated using the teacher network generated by machine learning using images obtained by connecting real environment images related to multiple modalities acquired at the same time and from the same direction in the channel direction as learning data. The information processing device according to (13). (15) The synthetic image is generated based on a process of reducing the distance in a feature space between a feature obtained by inputting the real environment image related to a certain modality into the teacher network and a feature obtained by inputting the real environment image related to a modality different from the certain modality into the teacher network. The information processing device according to (13). (16) The synthetic image is generated based on a process of converting a feature quantity obtained by inputting the real environment image related to a certain modality into a feature quantity obtained by inputting the real environment image related to a modality different from the certain modality into the teacher network. The information processing device according to (13). (17) an acquisition unit for acquiring images in the real environment; Further equipped with The estimation unit estimates an object class related to an object included in the image acquired by the acquisition unit. The information processing device according to any one of (1) to (16). (18) The acquisition unit acquires an image using at least one modality among a plurality of modalities used to acquire the real environment image used to generate the synthetic image. The information processing device according to (17). (19) A learning unit that generates a student network based on a teacher network generated by machine learning using images stored in a large-scale image database as training data; Equipped with the learning unit generates the student network by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation using the student network is expected to be performed as learning data; Information processing device. (20) Computer, an estimation unit that estimates an object class of an object included in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data; Equipped with The student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which the estimation by the estimation unit is expected to be performed, as learning data. Information processing device, A program to function as a [Explanation of symbols]

[0166] 10 First learning device 110 First Learning Section 20 Image generating device 210 Image Generation Unit 30 Second Learning Device 310 Third Learning Section 40 Estimation part 410 Acquisition Department 420 Estimation part 50 Large-scale database 60 Real Environment DB 70 Generated Image DB

Claims

1. an estimation unit that estimates an object class of an object included in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data; Equipped with The student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which the estimation by the estimation unit is expected to be performed, as learning data. Information processing device.

2. The synthetic image is generated based on adding noise to features obtained by inputting the real environment image to the teacher network. The information processing device according to claim 1 .

3. The synthetic image is generated based on adding noise in the principal component direction in a feature distribution obtained by inputting the real environment image to the teacher network. The information processing device according to claim 2 .

4. The synthetic image is generated so that the difference between a feature amount after average pooling obtained by inputting the real environment image to the teacher network and a feature amount obtained by adding noise to the feature amount in the principal component direction is small. The information processing device according to claim 3 .

5. The student network is generated by machine learning using the synthetic images whose similarity between different modalities exceeds a threshold among the generated synthetic images. The information processing device according to claim 1 .

6. The student network is generated by machine learning using the synthetic images whose similarity within the same modality exceeds a threshold among the generated synthetic images. The information processing device according to claim 1 .

7. The student network is generated by machine learning using a fusion image obtained by fusing a plurality of the generated synthetic images. The information processing device according to claim 1 .

8. The fusion image is generated by fusing a plurality of the synthetic images whose similarity exceeds a threshold value among the generated synthetic images. The information processing device according to claim 7.

9. The fusion image is generated by fusing a plurality of the synthetic images related to the same object class among the generated synthetic images. The information processing device according to claim 7.

10. The fusion image is generated by concatenating the synthetic images of multiple different modalities. The information processing device according to claim 7.

11. The fusion image is generated by concatenating, in a channel direction, a plurality of the synthetic images whose similarities between different modalities exceed a threshold value among the generated synthetic images. The information processing device according to claim 10.

12. The synthetic image is generated based on inputting the real environment image acquired by a corresponding modality to each of a plurality of teacher networks corresponding to a single modality. The information processing device according to claim 1 .

13. The synthetic image is generated based on a process of similarifying feature distributions related to each modality obtained by inputting the real environment image into the teacher network. The information processing device according to claim 1 .

14. The synthetic image is generated using the teacher network generated by machine learning using images obtained by connecting real environment images related to multiple modalities acquired at the same time and from the same direction in the channel direction as learning data. The information processing device according to claim 13.

15. The synthetic image is generated based on a process of reducing the distance in a feature space between a feature obtained by inputting the real environment image related to a certain modality into the teacher network and a feature obtained by inputting the real environment image related to a modality different from the certain modality into the teacher network. The information processing device according to claim 13.

16. The synthetic image is generated based on a process of converting a feature quantity obtained by inputting the real environment image related to a certain modality into a feature quantity obtained by inputting the real environment image related to a modality different from the certain modality into the teacher network. The information processing device according to claim 13.

17. an acquisition unit for acquiring images in the real environment; Further equipped with The estimation unit estimates an object class related to an object included in the image acquired by the acquisition unit. The information processing device according to claim 1 .

18. The acquisition unit acquires an image using at least one modality among a plurality of modalities used to acquire the real environment image used to generate the synthetic image. The information processing device according to claim 17.

19. A learning unit that generates a student network based on a teacher network generated by machine learning using images stored in a large-scale image database as training data; Equipped with the learning unit generates the student network by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which estimation using the student network is expected to be performed as learning data; Information processing device.

20. Computer, an estimation unit that estimates an object class of an object included in an input image by using a student network generated based on a teacher network generated by machine learning using images stored in a large-scale image database as learning data; Equipped with The student network is generated by machine learning using synthetic images obtained using the teacher network and real environment images acquired by a plurality of different modalities in a real environment in which the estimation by the estimation unit is expected to be performed, as learning data. Information processing device, A program to function as a

Citation Information

Patent Citations

  • Time-series information learning system, method, and neural network model

    JP2019159654A

  • Learning model evaluation method, learning method, device, and program

    JP2020004178A

  • Learning efficient object detection models with knowledge distillation

    US20180268292A1