Learning device, identification device, learning method, and learning program

Abstract

The missing data determination unit (7) determines whether there are mutually synchronized sample data for the first modality and sample data for the second modality. The loss calculation unit (discrimination loss calculation unit for modality 1 (10), discrimination loss calculation unit for modality 2 (13), and inter-feature distance loss calculation unit (14)) performs different loss calculations depending on whether there are mutually synchronized sample data for the first modality and sample data for the second modality, or whether there are no mutually synchronized sample data for the first modality and sample data for the second modality, and only one of the sample data exists. The model update unit (15) performs learning using the loss value obtained from the loss calculation by the loss calculation unit.

Description

Technical Field

[0001] This disclosure relates to machine learning using data of one or more modalities.

Background Art

[0002] Recently, multimodal learning has been spreading. Multimodal learning is machine learning using multimodal data. Multimodal data is data of two or more modalities. Multimodal learning can achieve higher accuracy than unimodal learning when specific unimodal data is input into a machine learning model during operation. Unimodal data is data of one modality. Also, unimodal learning is machine learning using unimodal data.

[0003] For example, in Patent Document 1, a multimodal learning method has been proposed in which learning is performed while reducing the distance between the feature amounts of the data of the modalities that can be used during learning and operation and the feature amounts of the data of the modalities that can be used only during learning. In addition, techniques that are robust to modality loss have been proposed when using multimodal data during learning and operation. For example, in Patent Document 2, a technique that can cope with the case where one modality is missing during operation due to a sensor failure or the like is disclosed.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The technology described in Patent Document 1, in the case of classification tasks such as image recognition, learns by bringing together the feature quantities of data from different modalities that have been assigned the same class label. In the technology described in Patent Document 1, when multiple labels are assigned to a single data point, such as in object detection or semantic occupancy prediction, the feature quantities of synchronized multimodal data can be brought together, and learning can be performed. However, in reality, the sampling frequencies of each sensor are often different. Therefore, data with missing data from one modality cannot be used for training as described in Patent Document 1.

[0006] The technology described in Patent Document 2 can handle modality loss during operation. However, it does not address how to handle data where modality is missing during training.

[0007] In light of these considerations, the primary objective of this disclosure is to enable effective learning whether or not there is sample data for two mutually synchronized modalities, or whether or not there is sample data for only one of them. [Means for solving the problem]

[0008] The learning device related to this disclosure is A sample data determination unit that determines whether or not there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality, Mutually synchronized sample data of the first modality and the second modality A loss calculation unit that performs different loss calculations depending on whether sample data exists or whether there is no mutually synchronized sample data for the first modality and the second modality, and only one of the sample data exists. The system includes a learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit. [Effects of the Invention]

[0009] This disclosure performs different loss calculations depending on whether there are sample data for two mutually synchronized modalities or only one sample data. Furthermore, this disclosure uses the loss values ​​obtained from the loss calculation for training. Therefore, according to this disclosure, effective training can be performed whether there are sample data for two mutually synchronized modalities or only one sample data. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing an example of the functional configuration of a multimodal learning device according to Embodiment 1. [Figure 2] A flowchart showing an example of operation of the multimodal learning device according to Embodiment 1. [Figure 3] A diagram showing an example of the functional configuration of a unimodal data identification device according to Embodiment 1. [Figure 4] A flowchart showing an example of operation of the unimodal data identification device according to Embodiment 1. [Figure 5] This figure shows an example of the functional configuration of a multimodal learning device for semantic occupation prediction according to Embodiment 2. [Figure 6] A diagram showing an example of a method for collecting image data and point cloud data according to Embodiment 2. [Figure 7] A diagram illustrating the calculation of the distance loss between feature quantities according to Embodiment 2. [Figure 8] A flowchart illustrating an example of operation of the multimodal learning device for semantic occupation prediction according to Embodiment 2. [Figure 9] This figure shows an example of the functional configuration of the image input semantic occupancy prediction device according to Embodiment 2. [Figure 10] A flowchart showing an example of operation of the image input semantic occupancy prediction device according to Embodiment 2. [Figure 11] This figure shows an example of the hardware configuration of a multimodal learning device according to Embodiment 1. [Figure 12] A diagram showing an example of the hardware configuration of a unimodal data identification device according to Embodiment 1.

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described with reference to the drawings. In the following description of the embodiments and the drawings, those denoted by the same reference numerals indicate the same or corresponding parts.

[0012] Embodiment 1. <Learning> ***Description of Configuration*** FIG. 1 shows a functional configuration example of a multimodal learning device 1 according to this embodiment. The multimodal learning device 1 is a learning device. Also, the operation procedure of the multimodal learning device 1 corresponds to a learning method. Further, a program for realizing the operation of the multimodal learning device 1 corresponds to a learning program. The multimodal learning device 1 is composed of a multimodal data collection unit 2, a data processing unit 3, a model unit 4, and a storage unit 5.

[0013] The multimodal data collection unit 2 collects two types of modal data from the multimodal data including images, point clouds, voices, characters, etc. The multimodal data collection unit 2 collects image data and point cloud data by, for example, one or more cameras and one or more LiDARs or millimeter-wave radars. LiDAR is Light Abbreviation for Detection and Ranging. The multimodal data collection unit 2 writes the collected data into the data storage unit 16. The data of the two modalities of the data collected by the multimodal data collection unit 2 are called modality 1 and modality 2. Modality 2 is data of a modality different from modality 1.

[0014] The data processing unit 3 processes modality 1 and modality 2 collected by the multimodal data collection unit 2. The data processing unit 3 is composed of an input processing unit 6 and a missing data determination unit 7.

[0015] The input processing unit 6 reads sample data included in the collected data from the data storage unit 16. Then, the input processing unit 6 outputs the read sample data to the missing data detection unit 7. Note that the sample data for Modality 1 corresponds to the sample data for the first modality, and the sample data for Modality 2 corresponds to the sample data for the second modality. In the following, sample data for Modality 1 will also be simply referred to as "samples for Modality 1." Similarly, sample data for Modality 2 will also be simply referred to as "samples for Modality 2."

[0016] The missing data detection unit 7 determines whether one of the modalities in the input sample data is missing. The missing data detection unit 7 then outputs the determination result to the input processing unit 6. The missing data detection unit 7 determines whether or not there is a sample of modality 2 that is synchronized with the sample of modality 1. The missing data detection unit 7 corresponds to the sample data detection unit. Furthermore, the processing performed by the missing data detection unit 7 corresponds to the sample data detection process.

[0017] The input processing unit 6 outputs sample data to the model unit 4 according to the determination result of the missing data determination unit 7. For example, if the missing data detection unit 7 determines that a sample for modality 2 is missing, the input processing unit 6 outputs a sample for modality 1 to the feature extraction unit 8 for modality 1. For example, if the missing data detection unit 7 determines that there are no missing sample data for both modalities, the input processing unit 6 outputs the sample data for modality 1 to the feature extraction unit 8 for modality 1 and the sample data for modality 2 to the feature extraction unit 11 for modality 2.

[0018] Model unit 4 updates the parameters of the machine learning model using multimodal data. Model Unit 4 consists of a feature extraction unit 8 for Modality 1, an identification unit 9 for Modality 1, an identification loss calculation unit 10 for Modality 1, a feature extraction unit 11 for Modality 2, an identification unit 12 for Modality 2, an identification loss calculation unit 13 for Modality 2, a feature distance loss calculation unit 14, and a model update unit 15.

[0019] The Modality 1 feature extraction unit 8 acquires a sample of Modality 1 and extracts the feature quantities of the Modality 1 sample as the first feature quantities. The Modality 1 feature extraction unit 8 then outputs the extracted Modality 1 sample feature quantities (first feature quantities) to the Modality 1 identification unit 9. The feature extraction unit 8 for modality 1 corresponds to the first feature quantity extraction unit.

[0020] The Modality 1 identification unit 9 acquires the feature quantities (first feature quantities) of the Modality 1 sample. The system then identifies the sample for Modality 1. The Modality 1 identification unit 9 outputs the identification result to the Modality 1 identification loss calculation unit 10. For example, in the case of object detection, the Modality 1 identification unit 9 identifies the class label and bounding box of the object included in the feature quantities of the Modality 1 sample. The identification unit 9 for modality 1 corresponds to the first identification unit.

[0021] The Modality 1 identification loss calculation unit 10 calculates the loss value using the identification results for the Modality 1 sample. In other words, the Modality 1 identification loss calculation unit 10 performs identification loss calculation using the Modality 1 sample.

[0022] The Modality 2 feature extraction unit 11 acquires a sample of Modality 2 and extracts the features of the Modality 2 sample as a second feature. The Modality 2 feature extraction unit 11 then outputs the extracted Modality 2 sample features (second feature) to the Modality 2 identification unit 12. The feature extraction unit 11 for modality 2 corresponds to the second feature extraction unit.

[0023] The Modality 2 identification unit 12 acquires the feature quantities (second feature quantities) of the Modality 2 samples and identifies the Modality 2 samples. The Modality 2 identification unit 12 outputs the identification result to the Modality 2 identification loss calculation unit 13. The identification unit 12 for modality 2 corresponds to the second identification unit.

[0024] The Modality 2 identification loss calculation unit 13 calculates the loss value using the identification results for the Modality 2 sample. In other words, the Modality 2 identification loss calculation unit 13 performs identification loss calculation using the Modality 2 sample.

[0025] The loss values ​​calculated by the Modality 1 identification loss calculation unit 10 and the Modality 2 identification loss calculation unit 13 represent the magnitude of the discrepancy between the predicted and actual values. The smaller the loss value, the more accurate the identification result.

[0026] The feature extraction unit 8 for modality 1, the identification unit 9 for modality 1, the feature extraction unit 11 for modality 2, and the identification unit 12 for modality 2 are composed of machine learning models such as Transformer and multilayer perceptron (MLP).

[0027] The feature-inter-feature distance loss calculation unit 14 acquires the feature quantities of the sample from modality 1 and the feature quantities of the sample from modality 2. Then, the feature-inter-feature distance loss calculation unit 14 calculates a loss value related to the distance between the two feature quantities. The smaller the loss value calculated by the feature distance loss calculation unit 14, the higher the similarity of the sample features between the two modalities. The feature distance loss calculation unit 14 calculates, for example, the cosine similarity between the two features (Cosine Calculate the value obtained by reversing the sign of the Similarity.

[0028] The discrimination loss calculation unit 10 for modality 1, the discrimination loss calculation unit 13 for modality 2, and the feature distance loss calculation unit 14 are collectively referred to as the loss calculation unit. As will be described later, the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized samples for modality 1 and modality 2, or whether there are no mutually synchronized samples for modality 1 and modality 2, and only one sample data exists. If there are synchronized samples from Modality 1 and Modality 2, the Modality 1 discrimination loss calculation unit 10 calculates the discrimination loss for the Modality 1 samples. The Modality 2 discrimination loss calculation unit 13 also calculates the discrimination loss for the Modality 2 samples. Furthermore, the feature-to-feature distance loss calculation unit 14 calculates the distance between the first and second features. We perform loss calculations for the distance. These loss calculations are collectively called compound loss calculations. On the other hand, if only samples from Modality 1 exist, the Modality 1 identification loss calculation unit 10 performs the identification loss calculation for the Modality 1 samples. Also, if only samples from Modality 2 exist, the Modality 2 identification loss calculation unit 13 performs the identification loss calculation for the Modality 2 samples. Either of these identification loss calculations is called a single-loss calculation.

[0029] The processing performed by the loss calculation unit (identification loss calculation unit 10 for modality 1, identification loss calculation unit 13 for modality 2, and feature distance loss calculation unit 14) corresponds to loss calculation processing.

[0030] The model update unit 15 performs learning using the loss values ​​obtained from the loss calculation performed by the loss calculation unit. In other words, if a compound loss calculation is performed, the model update unit 15 performs learning using the loss values ​​obtained from the compound loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 15 performs learning using the loss values ​​obtained from the single loss calculation. If a composite loss calculation is performed, the model update unit 15 learns the parameters of the Modality 1 feature extraction unit 8, the Modality 1 identification unit 9, the Modality 2 feature extraction unit 11, and the Modality 2 identification unit 12 to reduce the sum of the loss values ​​in the composite loss calculation. On the other hand, if a single-loss calculation is performed, the model update unit 15 learns the parameters of the feature extraction unit 8 for modality 1 or the feature extraction unit 11 for modality 2, and the parameters of the identification unit 9 for modality 1 or the identification unit 12 for modality 2, which reduce the loss value of the single-loss calculation. The model update unit 15 learns parameters, for example, by stochastic gradient descent. The model update unit 15 writes information about the updated parameters obtained through learning to the model storage unit 17. The model update unit 15 corresponds to the learning unit. Furthermore, the processing performed by the model update unit 15 corresponds to the learning process.

[0031] Memory unit 5 stores various types of information. The memory unit 5 consists of a data storage unit 16 and a model storage unit 17. The data storage unit 16 stores the data collected by the multimodal data acquisition unit 2. The model memory unit 17 stores the parameters of the model unit 4.

[0032] The multimodal learning device 1 is, for example, a computer having the hardware configuration shown in Figure 11.

[0033] The multimodal learning device 1 includes, as hardware, a processor 901, a main memory 902, an auxiliary memory 903, and a communication device 904. The functional components of Figure 1, excluding the memory unit 5, are implemented, for example, by a program. The auxiliary storage device 903 stores programs that implement these functions. These programs are loaded from the auxiliary storage device 903 into the main memory device 902. Then, the processor 901 executes these programs to perform the operation of the functional components shown in Figure 1. The memory unit 5 in Figure 1 is implemented, for example, by an auxiliary storage device 903.

[0034] ***Explanation of operation*** Next, we will explain the operation of the multimodal learning device 1. Figure 2 is a flowchart showing the learning operation by the multimodal learning device 1 according to Embodiment 1.

[0035] In step S101, the multimodal data acquisition unit 2 acquires data from modality 1 and modality 2. In step S101, it may be possible to acquire samples from modality 1 and modality 2 that are synchronized with each other, or it may be possible to acquire samples from only one modality because they are not synchronized with each other. The multimodal data acquisition unit 2 writes the collected data to the data storage unit 16.

[0036] In step S102, the loop begins.

[0037] In step S103, the input processing unit 6 reads sample data from the data storage unit 16.

[0038] In step S104, the missing data determination unit 7 determines whether the sample data read by the input processing unit 6 is sample data in which one of the modalities is missing. The input processing unit 6 selects a feature extraction unit based on the judgment result output by the missing data detection unit 7, and outputs the sample data to the selected feature extraction unit. Specifically, if there are synchronized samples from Modality 1 and Modality 2, the input processing unit 6 outputs the Modality 1 samples to the Modality 1 feature extraction unit 8. The input processing unit 6 also outputs the Modality 2 samples to the Modality 2 feature extraction unit 11 (the process proceeds to step S108). On the other hand, if there are no synchronized samples for modality 1 and modality 2, and only one sample data exists, the input processing unit 6 outputs the existing sample data to the corresponding feature extraction unit (the process proceeds to step S105). In other words, if only a sample for modality 1 exists, the input processing unit 6 outputs the sample for modality 1 to the feature extraction unit 8 for modality 1. If only a sample for modality 2 exists, the input processing unit 6 outputs the sample for modality 2 to the feature extraction unit 11 for modality 2.

[0039] In step S105, one of the feature extraction units extracts features from the sample data. In other words, if only Modality 1 samples are present, the Modality 1 feature extraction unit 8 extracts the features of the Modality 1 samples. On the other hand, if only Modality 2 samples are present, the Modality 2 feature extraction unit 11 extracts the features of the Modality 2 samples.

[0040] In step S106, one of the classifiers calculates a classification result using the features calculated in step S105. In other words, if feature quantities for a sample of modality 1 are extracted in step S105, the identification unit 9 for modality 1 uses the feature quantities for the sample of modality 1 to calculate the identification result for the sample of modality 1. On the other hand, if feature quantities for a sample of modality 2 are extracted in step S105, the identification unit 12 for modality 2 uses the feature quantities for the sample of modality 2 to calculate the identification result for the sample of modality 2.

[0041] In step S107, one of the identification loss calculation units calculates a loss value related to the identification result using the identification result calculated in step S106. In other words, if the identification result for the sample of modality 1 is calculated in step S106, the identification loss calculation unit 10 for modality 1 uses the identification result for the sample of modality 1 to perform an identification loss calculation for the identification result for the sample of modality 1 and calculates a loss value. On the other hand, if the identification result for the sample of modality 2 is calculated in step S106, the identification loss calculation unit 13 for modality 2 uses the identification result for the sample of modality 2 to perform an identification loss calculation for the identification result for the sample of modality 2. Perform the operation and calculate the loss value.

[0042] In step S108, the feature extraction unit 8 for modality 1 extracts feature quantities from the modality 1 sample. Additionally, the feature extraction unit 11 for modality 2 extracts feature quantities from the modality 2 sample.

[0043] In step S109, the feature distance loss calculation unit 14 calculates a loss value related to the distance between the feature of the sample from modality 1 and the feature of the sample from modality 2.

[0044] In step S110, the Modality 1 identification unit 9 calculates the identification result for the Modality 1 sample using the feature quantities of the Modality 1 sample. The Modality 2 identification unit 12 then calculates the identification result for the Modality 2 sample using the feature quantities of the Modality 2 sample.

[0045] In step S111, the identification loss calculation unit 10 for modality 1 uses the identification results for the sample of modality 1 to perform an identification loss calculation for the identification results for the sample of modality 1 and calculates a loss value. The identification loss calculation unit 13 for modality 2 also uses the identification results for the sample of modality 2 to perform an identification loss calculation for the sample of modality 2 and calculates a loss value.

[0046] In step S112, the model update unit 15 learns the parameters using the calculated loss and updates the parameters. More specifically, when a composite loss calculation is performed (steps S108 to S111), the model update unit 15 learns the parameters of the feature extraction unit 8 for modality 1, the parameters of the identification unit 9 for modality 1, the parameters of the feature extraction unit 11 for modality 2, and the parameters of the identification unit 12 for modality 2, which reduce the sum of the loss values ​​in the composite loss calculation. On the other hand, if a single loss calculation is performed (steps S105 to S107), the model update unit 15 learns the parameters of the feature extraction unit 8 for modality 1 or the feature extraction unit 11 for modality 2, and the parameters of the identification unit 9 for modality 1 or the identification unit 12 for modality 2, which reduce the loss value of the single loss calculation.

[0047] The process from step S102 to step S113 is repeated.

[0048] Finally, in step S114, the model update unit 15 writes information about the updated trained parameters to the model storage unit 17.

[0049] <Inference> ***Explanation of the structure*** Figure 3 shows an example of the functional configuration of the unimodal data identification device 18 according to this embodiment. The unimodal data identification device 18 is an identification device. The unimodal data identification device 18 is only capable of identifying data of modality 1. The unimodal data identification device 18 uses a machine learning model trained by the multimodal learning device 1 described above to identify data of modality 1.

[0050] The unimodal data identification device 18 consists of a data acquisition unit 19 for modality 1, an input processing unit 20, a model unit 21, an output processing unit 22, and a storage unit 23.

[0051] The Modality 1 data acquisition unit 19 collects data for Modality 1. The Modality 1 data acquisition unit 19 collects image data from, for example, one or more cameras as Modality 1 data. The data is written to the data storage unit 24.

[0052] The input processing unit 20 processes the image data collected by the data acquisition unit 19 for modality 1. The input processing unit 20 reads sample data contained in the collected image data from the data storage unit 24. Then, the input processing unit 20 outputs the read sample data to the model unit 21.

[0053] The model unit 21 calculates the identification result for the sample data of modality 1. The model unit 21 consists of a feature extraction unit 218 for modality 1 and an identification unit 219 for modality 1. The Modality 1 Feature Extraction Unit 218 reads the trained Modality 1 Feature Extraction Unit 8 parameters obtained by the Multimodal Learning Device 1 from the Model Storage Unit 17. Then, the Modality 1 Feature Extraction Unit 218 uses the read parameters to extract features from the Modality 1 sample data. The Modality 1 identification unit 219 reads the learned parameters of the Modality 1 identification unit 9 obtained by the multimodal learning device 1 from the model storage unit 17. Then, the Modality 1 identification unit 219 calculates the identification result for the sample data of Modality 1 using the read parameters.

[0054] The output processing unit 22 outputs the identification result of the Modality 1 identification unit 219. The output processing unit 22 displays the identification result on a display, for example.

[0055] The memory unit 23 stores various types of information. The memory unit 23 consists of a data memory unit 24 and a model memory unit 17. The data storage unit 24 stores the data collected by the modality 1 data acquisition unit 19. The model memory unit 17 stores information about the parameters of the model unit 4 that have been learned by the multimodal learning device 1.

[0056] The unimodal data identification device 18 is, for example, a computer having the hardware configuration shown in Figure 12.

[0057] The unimodal data identification device 18 includes, as hardware, a processor 911, a main memory 912, an auxiliary memory 913, and a communication device 914. The functional components in Figure 3, excluding the memory unit 23, are implemented, for example, by a program. The auxiliary storage device 913 stores programs that implement these functions. These programs are loaded from the auxiliary storage device 913 into the main memory device 912. Then, the processor 911 executes these programs to perform the operation of the functional components shown in Figure 3. The memory unit 23 in Figure 3 is implemented, for example, by an auxiliary storage device 913.

[0058] ***Explanation of operation*** Next, we will describe an example of the operation of the unimodal data identification device 18. In other words, we will describe the operation of the machine learning model trained by the multimodal learning device 1 during use. Figure 4 is a flowchart showing an example of the operation of the unimodal data identification device 18. As mentioned above, the unimodal data identification device 18 is only capable of identifying data of modality 1.

[0059] In step S201, the data acquisition unit 19 for modality 1 acquires the data for modality 1. The data collection unit 19 for Modality 1 collects the data and writes it to the data storage unit 24.

[0060] In step S202, the model unit 21 reads the trained parameters of the modality 1 feature extraction unit 8 and the trained parameters of the modality 1 discrimination unit 9 from the model storage unit 17.

[0061] In step S203, the input processing unit 20 reads sample data included in the data collected from the data storage unit 24. The input processing unit 20 outputs the read sample data to the feature extraction unit 218 for modality 1.

[0062] In step S204, the Modality 1 feature extraction unit 218 extracts features from the Modality 1 sample data using the parameters learned by the Modality 1 feature extraction unit 8. The Modality 1 feature extraction unit 218 outputs the extracted features to the Modality 1 discrimination unit 219.

[0063] In step S205, the Modality 1 identification unit 219 calculates the identification result for the Modality 1 sample data based on the features calculated by the Modality 1 feature extraction unit 218, using the learned parameters of the Modality 1 identification unit 9.

[0064] In step S206, the output processing unit 22 outputs the identification result for the sample data of modality 1.

[0065] ***Explanation of the effects of the embodiment*** In this embodiment, different loss calculations are performed depending on whether there are sample data from two mutually synchronized modalities or only one sample data. Then, in this embodiment, learning is performed using the loss value obtained from the loss calculation. Therefore, according to this embodiment, effective learning can be performed whether there are sample data from two mutually synchronized modalities or only one sample data.

[0066] Furthermore, in this embodiment, if one of the two modalities is missing, the sample data of the modality that is not missing is used to train the parameters of the corresponding feature extraction and discrimination units. This increases the amount of data that can be used as training data. As a result, the diversity of the training data increases, and a unimodal machine learning model with high generalization performance can be generated.

[0067] Furthermore, by making the features between modalities similar through the feature distance loss calculation unit, high-quality features that consider both modalities can be generated. Therefore, it is possible to generate models with higher accuracy than unimodal learning.

[0068] Furthermore, consider a scenario where one modality excels in the identification task, but only data from the other modality is available during operation. The multimodal learning device according to this embodiment can bring the feature vectors of the modality used during operation closer to the high-quality feature vectors of the modality that excels in the identification task. As a result, it becomes possible to generate high-quality feature vectors during operation, thereby improving the accuracy of the identification model during operation.

[0069] Embodiment 2. Embodiment 1 concerned a multimodal learning method and an operation method in a unimodal environment. In this embodiment, semantic occupancy prediction was used. This document explains multimodal learning methods and operational methods in a unimodal environment for NCY Prediction. This embodiment will primarily describe the differences from Embodiment 1. Matters not described below are the same as in Embodiment 1.

[0070] <Learning> ***Explanation of the structure*** Figure 5 shows an example of the functional configuration of the multimodal learning device 25 for semantic occupation prediction. The multimodal learning device 25 for semantic occupancy prediction is also a learning device. Furthermore, the operating procedure of the multimodal learning device 25 for semantic occupancy prediction corresponds to a learning method. Also, the program that implements the operation of the multimodal learning device 25 for semantic occupancy prediction corresponds to a learning program. The semantic occupancy prediction multimodal learning device 25 consists of a multimodal data acquisition unit 26, a data processing unit 27, a model unit 28, and a storage unit 29.

[0071] The multimodal data acquisition unit 26 acquires image data and point cloud data. For example, as shown in Figure 6, suppose that vehicle 81 is equipped with four RGB cameras 83, 84, 85, and 86. The multimodal data acquisition unit 26 collects image data from four viewpoints of the four RGB cameras 83, 84, 85, and 86. Also, suppose that vehicle 81 is equipped with one LiDAR 82. The multimodal data acquisition unit 26 collects point cloud data from the LiDAR 82. The multimodal data acquisition unit 26 may collect image data and point cloud data from robots, drones, etc., instead of the vehicle 81 in Figure 6. The number and position of cameras for collecting image data are not specified. Similarly, the number and position of LiDARs for collecting point cloud data are not specified. Millimeter-wave radar may also be used instead of LiDAR. Note that the sample data for collected image data and the sample data for point cloud data may not be synchronized. The multimodal data acquisition unit 26 writes the acquired data to the data storage unit 40.

[0072] The data processing unit 27 processes the data collected by the multimodal data acquisition unit 26. The data processing unit 27 consists of an input processing unit 30 and a missing data determination unit 31. The input processing unit 30 reads sample data included in the collected data from the data storage unit 40. Then, the input processing unit 30 outputs the read sample data to the missing data detection unit 31.

[0073] The missing data detection unit 31 determines whether one of the modalities in the input sample is missing. The missing data detection unit 31 then outputs the determination result to the input processing unit 30. The missing data determination unit 31 determines whether or not there is a point cloud sample data (also called a point cloud sample) that is synchronized with the image sample data (also called an image sample).

[0074] The input processing unit 30 outputs sample data to the model unit 28 according to the determination result of the missing data determination unit 31. For example, if the missing data detection unit 31 determines that a point cloud sample is missing, the input processing unit 30 outputs the image sample to the image feature extraction unit 32. For example, if the missing data detection unit 31 determines that there are no missing sample data for both modalities, the input processing unit 30 outputs the image sample to the image feature extraction unit 32 and the point cloud sample to the point cloud feature extraction unit 35.

[0075] The model unit 28 updates the parameters of the machine learning model using multimodal data. The model unit 28 consists of an image feature extraction unit 32, an image identification unit 33, an image identification loss calculation unit 34, a point cloud feature extraction unit 35, a point cloud identification unit 36, a point cloud identification loss calculation unit 37, a three-dimensional spatial feature representation distance loss calculation unit 38, and a model update unit 39.

[0076] The image feature extraction unit 32 acquires image samples and calculates a three-dimensional spatial feature representation of the image samples. The three-dimensional spatial feature representation holds information about the three-dimensional space to be reconstructed by semantic occupancy prediction. The three-dimensional spatial feature representation can be a voxel representation, a TPV (Tri-Perspective View) representation, a BEV (Bird's-Eye View) representation, etc. The image feature extraction unit 32 is not limited to these and can calculate feature quantities for any position in the three-dimensional space to be reconstructed as a three-dimensional spatial feature representation. Feature representations are included in the three-dimensional spatial feature representation. The image feature extraction unit 32 then extracts feature quantities from the calculated three-dimensional spatial feature representation of the image sample. The image feature extraction unit 32 corresponds to the first feature quantity extraction unit.

[0077] The image identification unit 33 obtains feature quantities from the three-dimensional spatial feature representation of the image sample and identifies the image sample. The image identification unit 33 outputs the identification result to the image identification loss calculation unit 34. The image identification unit 33 corresponds to the first identification unit.

[0078] The image recognition loss calculation unit 34 calculates a loss value using the recognition result for the image sample.

[0079] The point cloud feature extraction unit 35 acquires point cloud samples and calculates the three-dimensional spatial feature representation of the point cloud samples. The point cloud feature extraction unit 35 then extracts feature quantities from the calculated three-dimensional spatial feature representation of the point cloud sample. The point cloud feature extraction unit 35 corresponds to the second feature quantity extraction unit.

[0080] The point cloud identification unit 36 ​​acquires feature quantities obtained from the three-dimensional spatial feature representation of the point cloud sample and identifies the point cloud sample. The point cloud identification unit 36 ​​outputs the identification result to the point cloud identification loss calculation unit 37. The point cloud identification unit 36 ​​corresponds to the second identification unit.

[0081] The point cloud identification loss calculation unit 37 calculates the loss value using the identification result for the point cloud sample.

[0082] The image recognition loss calculation unit 34 and the point cloud recognition loss calculation unit 37 calculate loss values ​​such that the smaller the value, the more accurate the recognition result.

[0083] The image feature extraction unit 32, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​are composed of machine learning models such as Transformer or multilayer perceptron (MLP).

[0084] As shown in Figure 7, the three-dimensional spatial feature representation distance loss calculation unit 38 calculates a loss value related to the distance between two feature quantities for the same three-dimensional position feature quantities between the modalities for the three-dimensional spatial feature representations of both modalities. Figure 7 illustrates the calculation of the distance loss between features. Figure 7 shows the three-dimensional spatial feature tables of both modalities. While the current representation is voxel representation, the same applies when using other three-dimensional spatial feature representations. Different three-dimensional spatial feature representations may be used between modalities. The loss value calculated by the three-dimensional spatial feature representation distance loss calculation unit 38 indicates that the smaller the value, the higher the similarity of the feature quantities of the samples between the two modalities.

[0085] The image recognition loss calculation unit 34, the point cloud recognition loss calculation unit 37, and the three-dimensional spatial feature representation distance loss calculation unit 38 are collectively referred to as the loss calculation unit. As will be described later, the loss calculation unit performs different loss calculations depending on whether there are mutually synchronized image samples and point cloud samples, or whether there are no mutually synchronized image samples and point cloud samples and only one of the sample data exists. If there are mutually synchronized image samples and point cloud samples, the image discrimination loss calculation unit 34 calculates the discrimination loss of the image samples. The point cloud discrimination loss calculation unit 37 also calculates the discrimination loss of the point cloud samples. Furthermore, the three-dimensional spatial feature representation distance loss calculation unit 38 calculates the loss for the distance between the feature quantities of the image samples and the feature quantities of the point cloud samples. These loss calculations are collectively referred to as combined loss calculations. On the other hand, if only image samples exist, the image recognition loss calculation unit 34 performs the recognition loss calculation for the image samples. If only point cloud samples exist, the point cloud recognition loss calculation unit 37 performs the recognition loss calculation for the point cloud samples. Either of these recognition loss calculations is called a single-loss calculation.

[0086] The processing performed by the loss calculation unit (image identification loss calculation unit 34, point cloud identification loss calculation unit 37, and three-dimensional spatial feature representation distance loss calculation unit 38) corresponds to loss calculation processing.

[0087] The model update unit 39 performs learning using the loss values ​​obtained by the loss calculation unit. If a compound loss calculation is performed, the model update unit 39 performs learning using the loss values ​​obtained by the compound loss calculation. On the other hand, if a single loss calculation is performed, the model update unit 39 performs learning using the loss values ​​obtained by the single loss calculation. If a composite loss calculation is performed, the model update unit 39 learns the parameters of the image feature extraction unit 32, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​to reduce the sum of the loss values ​​in the composite loss calculation. On the other hand, if a single-loss calculation is performed, the model update unit 39 learns the parameters of the image feature extraction unit 32 or the point cloud feature extraction unit 35, and the parameters of the image identification unit 33 or the point cloud identification unit 36, which reduce the loss value of the single-loss calculation. The model update unit 39 writes information about the updated parameters obtained through learning to the model storage unit 41. The model update unit 39 corresponds to the learning unit. Furthermore, the processing performed by the model update unit 39 corresponds to the learning process.

[0088] The memory unit 29 stores various types of information. The memory unit 29 consists of a data memory unit 40 and a model memory unit 41. The data storage unit 40 stores the data collected by the multimodal data acquisition unit 26. The model memory unit 41 stores the parameters of the model unit 28.

[0089] The multimodal learning device 25 for semantic occupation prediction is, for example, a computer having the hardware configuration shown in Figure 11.

[0090] The multimodal learning device 25 for semantic occupancy prediction includes, as hardware, a processor 901, a main memory 902, an auxiliary memory 903, and a communication device 904. The functional components of Figure 5, excluding the memory unit 29, are implemented, for example, by a program. The auxiliary storage device 903 stores programs that implement these functions. These programs are loaded from the auxiliary storage device 903 into the main memory device 902. Then, the processor 901 executes these programs to perform the operation of the functional components shown in Figure 5. The memory unit 29 in Figure 5 is implemented, for example, by an auxiliary storage device 903.

[0091] ***Explanation of operation*** Next, the operation of the multimodal learning device 25 for semantic occupation prediction will be described. Figure 8 is a flowchart showing the learning operation by the multimodal learning device 25 for semantic occupation prediction according to Embodiment 2.

[0092] In step S301, the multimodal data acquisition unit 262 acquires image data and point cloud data. In step S301, there are cases where synchronized image samples and point cloud samples can be acquired, and cases where samples from only one modality can be acquired because they are not synchronized. The multimodal data acquisition unit 26 writes the acquired data to the data storage unit 40.

[0093] In step S302, the loop is initiated.

[0094] In step S303, the input processing unit 30 reads sample data from the data storage unit 40.

[0095] In step S304, the missing data determination unit 31 determines whether the sample read by the input processing unit 30 is a sample in which one of the modalities is missing. The input processing unit 30 selects a feature extraction unit based on the judgment result output by the missing data determination unit 31, and outputs the sample data to the selected feature extraction unit. Specifically, if there are mutually synchronized image samples and point cloud samples, the input processing unit 30 outputs the image sample to the image feature extraction unit 32. The input processing unit 30 also outputs the point cloud sample to the point cloud feature extraction unit 35 (the process proceeds to step S308). On the other hand, if there are no mutually synchronized image samples and point cloud samples, and only one sample data exists, the input processing unit 30 outputs the existing sample data to the corresponding feature extraction unit (the process proceeds to step S305). In other words, if only an image sample exists, the input processing unit 30 outputs the image sample to the image feature extraction unit 32. If only a point cloud sample exists, the input processing unit 30 outputs the point cloud sample to the point cloud feature extraction unit 35.

[0096] In step S305, one of the feature extraction units calculates a three-dimensional spatial feature representation of the sample data. In other words, if only an image sample exists, the image feature extraction unit 32 calculates the three-dimensional spatial feature representation of the image sample. On the other hand, if only a point cloud sample exists, the point cloud feature extraction unit 35 calculates the three-dimensional spatial feature representation of the point cloud sample.

[0097] In step S306, one of the identification units calculates an identification result using the features obtained from the three-dimensional spatial feature representation calculated in step S305. In other words, if feature quantities for the 3D spatial feature representation of an image sample are extracted in step S305, the image identification unit 33 calculates an identification result for the image sample using the feature quantities of the image sample. On the other hand, if feature quantities for the 3D spatial feature representation of a point cloud sample are extracted in step S305, the point cloud identification unit 36 ​​calculates an identification result for the point cloud sample using the feature quantities of the point cloud sample.

[0098] In step S307, using the identification result calculated in step S306, one of the The identification loss calculation unit calculates a loss value related to the identification result. In other words, if an identification result for an image sample is calculated in step S306, the image identification loss calculation unit 34 calculates a loss value related to the identification result for the image sample. On the other hand, if an identification result for a point cloud sample is calculated in step S306, the point cloud identification loss calculation unit 37 calculates a loss value related to the identification result for the point cloud sample.

[0099] In step S308, the image feature extraction unit 32 calculates the three-dimensional spatial feature representation of the image sample. The point cloud feature extraction unit 35 also calculates the three-dimensional spatial feature representation of the point cloud sample.

[0100] In step S309, the three-dimensional spatial feature representation distance loss calculation unit 38 calculates a loss value related to the distance between the three-dimensional spatial feature representation of the image sample and the three-dimensional spatial feature representation of the point cloud sample, for the same three-dimensional position feature quantities.

[0101] In step S310, the image identification unit 33 calculates an identification result for the image sample using the feature quantities of the image sample. The point cloud identification unit 36 ​​also calculates an identification result for the point cloud sample using the feature quantities of the point cloud sample.

[0102] In step S311, the image recognition loss calculation unit 34 calculates a loss value related to the recognition result for the image sample using the recognition result for the image sample. The image recognition loss calculation unit 34 also calculates a loss value related to the recognition result for the point cloud sample using the recognition result for the point cloud sample.

[0103] In step S312, the model update unit 39 learns the parameters using the calculated loss and updates the parameters. More specifically, when a composite loss calculation is performed (steps S308 to S311), the model update unit 39 learns the parameters of the image feature extraction unit 328, the image identification unit 33, the point cloud feature extraction unit 35, and the point cloud identification unit 36 ​​to reduce the sum of the loss values ​​in the composite loss calculation. On the other hand, if a single-loss calculation is performed (steps S305 to S307), the model update unit 15 learns the parameters of the image feature extraction unit 32 or the point cloud feature extraction unit 35, and the parameters of the image identification unit 33 or the point cloud identification unit 36, which reduce the loss value of the single-loss calculation.

[0104] The process from step S302 to step S113 is repeated.

[0105] Finally, in step S314, the model update unit 39 writes information about the updated trained parameters to the model storage unit 41.

[0106] <Inference> ***Explanation of the structure*** Figure 9 shows an example of the functional configuration of the image input semantic occupancy prediction device 42 according to this embodiment. The image input semantic occupancy prediction device 42 is an identification device. The image input semantic occupancy prediction device 42 is only capable of identifying image data. The image input semantic occupancy prediction device 42 uses a machine learning model trained by the aforementioned multimodal learning device 25 for semantic occupancy prediction to identify image data.

[0107] The image input semantic occupancy prediction device 42 consists of an image data acquisition unit 43, an input processing unit 44, a model unit 45, an output processing unit 46, and a storage unit 47.

[0108] The image data collection unit 43 collects image data. The image data acquisition unit 43 collects image data, for example, from one or more cameras. The image data acquisition unit 43 writes the collected image data to the data storage unit 24.

[0109] The input processing unit 44 processes the image data collected by the image data collection unit 43. The input processing unit 44 reads sample data contained in the collected image data from the data storage unit 24. Then, the input processing unit 44 outputs the read sample data to the model unit 45.

[0110] The model unit 45 calculates the identification result for the sample image data. The model unit 45 consists of an image feature extraction unit 452 and an image identification unit 453. The image feature extraction unit 452 reads the trained parameters of the image feature extraction unit 32 obtained by the semantic occupancy prediction multimodal learning device 25 from the model storage unit 41. Then, the image feature extraction unit 452 extracts feature quantities from the sample image data using the read parameters. The image recognition unit 453 reads the learned parameters of the image recognition unit 33 obtained by the semantic occupancy prediction multimodal learning device 25 from the model storage unit 41. Then, the image recognition unit 453 calculates the recognition result for the sample image data using the read parameters.

[0111] The output processing unit 46 outputs the identification result of the image identification unit 453. The output processing unit 46 displays the identification result on a display, for example.

[0112] The memory unit 47 stores various types of information. The memory unit 47 consists of a data memory unit 48 and a model memory unit 41. The data storage unit 48 stores the data collected by the image data acquisition unit 43. The model memory unit 41 stores information about the parameters of the model unit 4 that have been learned by the semantic occupation prediction multimodal learning device 25.

[0113] The image input semantic occupancy prediction device 42 is, for example, a computer having the hardware configuration shown in Figure 12.

[0114] The image input semantic occupancy prediction device 42 includes, as hardware, a processor 911, a main memory 912, an auxiliary memory 913, and a communication device 914. The functional components in Figure 9, excluding the memory unit 47, are implemented, for example, by a program. The auxiliary storage device 913 stores programs that implement these functions. These programs are loaded from the auxiliary storage device 913 into the main memory device 912. Then, the processor 911 executes these programs to perform the operation of the functional components shown in Figure 9. The memory unit 47 in Figure 9 is implemented, for example, by an auxiliary storage device 913.

[0115] ***Explanation of operation*** Next, we will describe an example of the operation of the image input semantic occupancy prediction device 42. In other words, we will describe the operation of the machine learning model trained by the semantic occupancy prediction multimodal learning device 25 during operation. Figure 10 is a flowchart showing an example of the operation of the image input semantic occupancy prediction device 42. As mentioned above, the image input semantic occupancy prediction device 42 is only capable of identifying image data.

[0116] In step S401, the image data acquisition unit 43 acquires image data. The image data acquisition unit 43 writes the acquired data to the data storage unit 48.

[0117] In step S402, the model unit 45 reads the trained parameters of the image feature extraction unit 32 and the trained parameters of the image recognition unit 33 from the model storage unit 41.

[0118] In step S403, the input processing unit 44 reads sample data included in the data collected from the data storage unit 48. The input processing unit 44 outputs the read sample data to the image feature extraction unit 32.

[0119] In step S404, the image feature extraction unit 452 calculates a three-dimensional spatial feature representation of the image sample using the parameters learned by the image feature extraction unit 32. The image feature extraction unit 32 then extracts features from the three-dimensional spatial feature representation. The image feature extraction unit 32 outputs the extracted features to the image identification unit 453.

[0120] In step S405, the image identification unit 453 calculates an identification result for an image sample based on the feature quantities calculated by the image feature extraction unit 452, using the learned parameters of the image identification unit 33.

[0121] In step S406, the output processing unit 46 outputs the identification result for the image sample.

[0122] ***Effects of the Embodiment*** In Embodiment 2, in addition to the effects of Embodiment 1, it is possible to sample a certain number of three-dimensional positions used to calculate the distance between three-dimensional spatial feature representations. This makes it possible to achieve multimodal learning with improved computational efficiency.

[0123] Although Embodiments 1 and 2 have been described above, these two embodiments may be implemented in combination. Alternatively, one of these two embodiments may be partially implemented. Alternatively, these two embodiments may be partially combined and implemented. Furthermore, the configurations and procedures described in these two embodiments may be modified as needed.

[0124] ***Supplementary explanation of hardware configuration*** Here, we will provide supplementary explanations for the hardware configuration shown in Figures 11 and 12. The following explanation will provide supplementary details for the hardware configuration shown in Figure 11, but the same explanation applies to the hardware configuration shown in Figure 12.

[0125] Processor 901 includes CPU, DSP, etc. CPU stands for Central Processing Unit. DSP stands for Digital Signal Processor. Main memory 902 is RAM. RAM stands for Random Access Memory. The auxiliary storage device 903 includes ROM, flash memory, HDD, etc. ROM stands for Read Only Memory. HDD stands for Hard Disk Drive. The communication device 904 is an electronic circuit that performs data communication processing. The communication device 904 is, for example, a communication chip or a NIC. NIC stands for Network Interface Card.

[0126] Furthermore, the auxiliary storage device 903 also stores the OS. OS stands for Operating System. Then, at least a portion of the OS is executed by processor 901. The processor 901 executes programs that realize the functions of the functional components shown in Figure 1 or Figure 5, while executing at least a part of the OS. Processor 901 executes the OS, handling task management, memory management, file management, communication control, and other functions. Furthermore, at least one of the information, data, signal values, and variable values ​​indicating the processing results of the functional components shown in Figure 1 or Figure 5 is stored in at least one of the main memory 902, auxiliary memory 903, registers in the processor 901, and cache memory. Furthermore, the programs that realize the functions of the functional components shown in Figure 1 or Figure 5 may be stored on portable recording media such as magnetic disks, flexible disks, optical disks, compact disks, Blu-ray® discs, and DVDs. Portable recording media containing programs that realize the functions of the functional components shown in Figure 1 or Figure 5 may also be distributed.

[0127] Furthermore, at least one of the "parts" in the functional components shown in Figure 1 or Figure 5 may be read as "circuit," "process," "procedure," "process," or "circuitry." Furthermore, the multimodal learning device 1 and the multimodal learning device 25 for semantic occupancy prediction may each be implemented by processing circuits. Examples of processing circuits include logic ICs, GAs, ASICs, and FPGAs. IC stands for Integrated Circuit. GA stands for Gate Array. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field-Programmable Gate Array. In this case, the functional components shown in Figure 1 or Figure 5 are each implemented as part of the processing circuit. In this specification, the higher-level concept encompassing both the processor and the processing circuit is referred to as "processing circuitry." In other words, a processor and a processing circuit are specific examples of "processing circuits," respectively.

[0128] Finally, the various aspects of this disclosure are summarized below as an appendix. (Note 1) A sample data determination unit that determines whether or not there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a modality different from the first modality, A loss calculation unit that performs different loss calculations depending on whether there are mutually synchronized sample data for the first modality and the second modality, or whether there are no mutually synchronized sample data for the first modality and the second modality, and only one of the sample data exists. A learning device having a learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit. (Note 2) The loss calculation unit, When there are mutually synchronized sample data for the first modality and sample data for the second modality, a composite loss calculation is performed using the mutually synchronized sample data for the first modality and sample data for the second modality. Mutually synchronized sample data of the first modality and the second modality If no sample data exists, and only one sample data exists, a single loss calculation is performed using the existing sample data. The aforementioned learning unit, The learning device described in Appendix 1, which, when the composite loss calculation is performed by the loss calculation unit, performs learning using the loss value obtained by the composite loss calculation, and when the single loss calculation is performed by the loss calculation unit, performs learning using the loss value obtained by the single loss calculation. (Note 3) The loss calculation unit, As part of the composite loss calculation, the discrimination loss calculation for the sample data of the first modality, the discrimination loss calculation for the sample data of the second modality, and the loss calculation for the distance between the feature quantities of the sample data of the first modality and the feature quantities of the sample data of the second modality are performed. The learning device described in Appendix 2, which performs, as the single-loss calculation, the discrimination loss calculation of sample data for the first modality or the discrimination loss calculation of sample data for the second modality. (Appendix 4) The learning device further, A first feature extraction unit extracts the features of the sample data of the first modality as the first feature, A second feature extraction unit extracts the features of the sample data of the second modality as the second feature, A first identification unit that uses the first feature quantity to identify sample data of the first modality, It includes a second identification unit that uses the second feature quantity to identify sample data of the second modality, The loss calculation unit, As part of the composite loss calculation, the following are performed: an identification loss calculation for the identification result by the first identification unit, an identification loss calculation for the identification result by the second identification unit, and a loss calculation for the distance between the first feature and the second feature. The learning device described in Appendix 3, which performs, as the single loss calculation, an identification loss calculation for the identification result by the first identification unit or an identification loss calculation for the identification result by the second identification unit. (Note 5) The aforementioned learning unit, If the aforementioned composite loss calculation is performed, the parameters of the first feature extraction unit, the parameters of the second feature extraction unit, the parameters of the first discrimination unit, and the parameters of the second discrimination unit are learned to reduce the loss value obtained by the aforementioned composite loss calculation. The learning device described in Appendix 4, which, when the single-loss calculation is performed, learns the parameters of the first feature extraction unit or the second feature extraction unit and the parameters of the first identification unit or the second identification unit to reduce the loss value obtained by the single-loss calculation. (Note 6) The aforementioned sample data determination unit is: It is determined whether there are mutually synchronized image sample data, which is sample data for the first modality, and point cloud sample data, which is sample data for the second modality. The loss calculation unit, Different loss calculations are performed depending on whether there is mutually synchronized image sample data and point cloud sample data, or whether there is no mutually synchronized image sample data and point cloud sample data, and only one of the sample data exists. A learning device as described in any one of the appendices 1 to 5. (Note 7) An identification device that uses the learning results from the learning unit described in Appendix 1 to identify data of the first modality or data of the second modality. (Note 8) The computer determines whether there are sample data for a first modality that are synchronized with each other, and sample data for a second modality that is a different modality from the first modality. The computer performs different loss calculations depending on whether there are mutually synchronized sample data for the first modality and the second modality, or whether there are no mutually synchronized sample data for the first modality and the second modality, and only one of the sample data sets exists. A learning method in which the computer performs learning using the loss value obtained by the loss calculation. (Note 9) A sample data determination process that determines whether there are sample data of a first modality that are synchronized with each other and sample data of a second modality that is a different modality from the first modality, A loss calculation process that performs different loss calculations depending on whether there are mutually synchronized sample data for the first modality and the second modality, or whether there are no mutually synchronized sample data for the first modality and the second modality, and only one of the sample data exists. A learning program having a learning process that performs learning using the loss value obtained by the loss calculation process described above. [Explanation of Symbols]

[0129] 1 Multimodal learning device, 2 Multimodal data acquisition unit, 3 Data processing unit, 4 Model unit, 5 Storage unit, 6 Input processing unit, 7 Missing data detection unit, 8 Feature extraction unit for Modality 1, 9 Identification unit for Modality 1, 10 Identification loss calculation unit for Modality 1, 11 Feature extraction unit for Modality 2, 12 Identification unit for Modality 2, 13 Identification loss calculation unit for Modality 2, 14 Inter-feature distance loss calculation unit, 15 Model update unit, 16 Data storage unit, 17 Model storage unit, 18 Unimodal data identification device, 19 Data acquisition unit for Modality 1, 20 Input processing unit, 21 Model unit, 218 Feature extraction unit for Modality 1, 219 Identification unit for Modality 1, 22 Output processing unit, 23 Storage unit, 24 Data storage unit, 25 Multimodal learning device for semantic occupancy prediction, 26 Multimodal data acquisition unit, 27 1. Data processing unit, 28. Model unit, 29. Storage unit, 30. Input processing unit, 31. Missing data detection unit, 32. Image feature extraction unit, 33. Image identification unit, 34. Image identification loss calculation unit, 35. Point cloud feature extraction unit, 36. Point cloud identification unit, 37. Point cloud identification loss calculation unit, 38. Three-dimensional spatial feature representation distance loss calculation unit, 39. Model update unit, 40. Data storage unit, 41. Model storage unit, 42. Image input semantic occupancy prediction device, 43. Image data collection unit, 44. Input processing unit, 45. Model unit, 452. Image feature extraction unit, 453. Image identification unit, 46. Output processing unit, 47. Storage unit, 48. Data storage unit, 901. Processor, 902. Main memory, 903. Auxiliary storage, 904. Communication device, 911. Processor, 912. Main memory, 913. Auxiliary storage, 914. Communication device.

Claims

1. Mutually synchronized sample data of the first modality and data different from the first modality This determines whether or not sample data for the second modality exists. Sample data determination unit, Mutually synchronized sample data of the first modality and the second modality When sample data exists, and the samples of the first modality are synchronized with each other If sample data for the second modality is not available, and either one of the samples is not available... A loss calculation unit that performs different loss calculations depending on whether only data is present or not. A learning unit that performs learning using the loss value obtained by the loss calculation performed by the loss calculation unit. A learning device having the following features.

2. The loss calculation unit, Mutually synchronized sample data of the first modality and the second modality If sample data exists, the samples of the first modality that are synchronized with each other A composite loss meter is a loss calculation that uses sample data and sample data of the second modality. Perform the calculation, Mutually synchronized sample data of the first modality and the second modality If no sample data exists, and only one sample data exists: Then, a single loss calculation, which is a loss calculation using existing sample data, is performed. The aforementioned learning unit, If the composite loss calculation is performed by the loss calculation unit, the result obtained from the composite loss calculation Learning is performed using the obtained loss value, and the single loss calculation is performed by the loss calculation unit. The learning method according to claim 1 is performed using the loss value obtained by the single loss calculation. Device.

3. The loss calculation unit, The aforementioned combined loss calculation includes the identification loss calculation of the sample data for the first modality, The recognition loss calculation of the sample data of the second modality and the sample data of the first modality The distance between the features of the pull data and the features of the sample data of the second modality is... We will perform a loss calculation, As the single loss calculation, the identification loss calculation of the sample data of the first modality or The learning apparatus according to claim 2, which performs the identification loss calculation of sample data of the second modality. 。

4. The learning device further, The first method extracts the features of the sample data of the first modality as the first feature. Feature extraction unit, The second method extracts the features of the sample data for the second modality as the second feature. Feature extraction unit, A first method for identifying sample data of the first modality using the first feature quantity. Identification unit, The second feature is used to identify the sample data of the second modality. It has an identification unit, The loss calculation unit, The aforementioned composite loss calculation includes an identification loss calculation for the identification result by the first identification unit and , the calculation of the identification loss for the identification result by the second identification unit, and the first feature quantity and the We perform a loss calculation on the distance between the second feature and the first feature. As the single loss calculation, the identification loss calculation for the identification result by the first identification unit or The learning according to claim 3, which performs an identification loss calculation for the identification result by the second identification unit. Device.

5. The aforementioned learning unit, If the above-mentioned composite loss calculation is performed, the loss value obtained by the above-mentioned composite loss calculation is reduced. The parameters of the first feature extraction unit and the parameters of the second feature extraction unit The parameters of the first identification unit and the parameters of the second identification unit are learned. If the single loss calculation described above is performed, the loss value obtained by the single loss calculation is reduced. The parameters of the first feature extraction unit or the parameters of the second feature extraction unit The claim is to learn the parameters of the first identification unit or the parameters of the second identification unit. The learning device described in 4.

6. The aforementioned sample data determination unit is: Image sample data, which is sample data of the first modality that is synchronized with each other. Whether or not point cloud sample data, which is sample data for the second modality, exists. Determine, The loss calculation unit, In the case where the aforementioned image sample data and point cloud sample data are synchronized with each other If the image sample data and point cloud sample data are synchronized with each other, First, different loss calculations are performed depending on whether only one sample data point exists or not. A learning device according to any one of claims 1 to 5.

7. Using the learning results from the learning unit described in claim 1, the data of the first modality Or an identification device for identifying data of the second modality.

8. The computer uses sample data of the first modality that is synchronized with each other and the first Are there sample data for a second modality that is different from the first modality? Determine whether it is true or false. The computer then synchronizes the sample data of the first modality with the previous When sample data for the second modality exists, and the first modality is synchronized with each other Sample data for the first modality and sample data for the second modality do not exist. Different loss calculations are performed depending on whether only one sample data is available or not. A learning method in which the computer performs learning using the loss value obtained by the loss calculation. 。

9. Mutually synchronized sample data of the first modality and data different from the first modality This determines whether or not sample data for the second modality exists. Sample data determination process, Mutually synchronized sample data of the first modality and the second modality When sample data exists, and the samples of the first modality are synchronized with each other If sample data for the second modality is not available, and either one of the samples is not available... Loss calculation processes that perform different loss calculations depending on whether only data is available or not. A learning process that performs learning using the loss value obtained by the loss calculation performed by the loss calculation process described above. A learning program that uses logic to enable a computer to execute instructions.

Images