Generation of descriptors for data of the second modality associated with data of the first modality.

Abstract

The present invention relates to a computer-implemented method of generating a descriptor associated with data of a first modality, the method comprising the steps of receiving first data associated with a first modality and second data related to a second modality, the first and second modalities being different, generating respective first and second descriptors corresponding to the respective first and second modalities by encoding the first and second data using respective first and second encoders, the first and second encoders being trained based on first training data of the first modality and second training data of the second modality, respectively, converting the first descriptor into a third descriptor, the third descriptor corresponding to the second modality, and storing the third descriptor in a database.

Description

Technical Field

[0001] The subject matter of the present disclosure relates to a computer-implemented method for generating a descriptor of data of a first modality, a computer-implemented method for training a descriptor generator for generating a descriptor of data of a first modality, a computer-implemented method for retrieving data of a first modality, and a non-transitory or transitory computer-readable medium.

Background Art

[0002] An autonomous vehicle (AV) includes various sensors of different sensor types. Each sensor type records data of a different modality. Different modalities can be paired when captured simultaneously by sensors on a particular AV. For example, the paired data may be data of two or more modalities that captured the same scene. The paired data can be labeled as such. The paired data can be retrieved for various purposes, such as comparing with current data to control the AV.

Summary of the Invention

Problems to be Solved by the Invention

[0003] However, some AVs may not capture paired data. In such cases, if data of different modalities are individually labeled, it is impossible to identify which data corresponds to which scene, so data retrieval becomes a problem.

[0004] It is an object of the present disclosure to address such problems and improve the prior art.

Means for Solving the Problems

[0005] According to one embodiment, a computer implementation method is provided for generating descriptors associated with data of a first modality. This method includes receiving first data associated with a first modality and second data associated with a second modality, wherein the first and second modalities are different; generating by encoding the first and second data using the first and second encoders, respectively, the first and second encoders being trained on first training data of the first modality and second training data of the second modality, respectively; converting the first descriptors into third descriptors, wherein the third descriptors correspond to the second modality; and storing the third descriptors in a database.

[0006] The above-described embodiment may be expressed differently as a computer implementation method for generating descriptors associated with data of a first modality recorded by a sensor of a first modality, wherein the autonomous vehicle is equipped with a sensor of a first modality and a sensor of a second modality, and the method comprises the steps of recording first data associated with the first modality by the sensor of the first modality and recording second data associated with the second modality by the sensor of the second modality, wherein the first and second modalities are different, and the first and second data respectively The process includes generating first and second descriptors corresponding to each first and second modality by encoding them using first and second encoders, wherein the first and second encoders are trained on first training data of the first modality and second training data of the second modality, respectively; converting the first descriptors to third descriptors corresponding to the second modality; and storing the third descriptors in a database.

[0007] In one embodiment, the conversion step may include decoding a first descriptor into decoded third data corresponding to a second modality using a decoder, the decoder being trained to generate data for the second modality, and encoding the third data to generate a third descriptor corresponding to the second modality using a third encoder.

[0008] In one embodiment, the first modality and the second modality may correspond to the modality of the sensor used to record their respective data.

[0009] In one embodiment, the sensor may include one of a LiDAR sensor, a RADAR sensor, and a camera. In other words, the sensor of the first modality and the sensor of the second modality may each include one of a LiDAR sensor, a RADAR sensor, and a camera.

[0010] In one embodiment, the first encoder and the second encoder may be from their respective autoencoders, and the first descriptor and the second descriptor may be representations corresponding to the bottleneck of their respective autoencoders.

[0011] In one embodiment, the autoencoder may be a variational autoencoder.

[0012] According to another aspect of the present invention, a computer implementation method is provided for training a descriptor generator for generating descriptors associated with data of a first modality, the descriptor generator comprising a first encoder, a second encoder, and a transformer. The method includes the steps of: training a first encoder to generate a first descriptor corresponding to first data of a first modality using first training data of a first modality; training a second encoder to generate a second descriptor corresponding to second data of a second modality using second training data of a second modality, wherein the first and second modalities are different; and training a transformer to transform the first descriptor into a third descriptor corresponding to a second modality using comparative learning based on the first and second training data.

[0013] In one embodiment, the step of training a transformer using comparative learning may include the steps of determining the distance between a second descriptor and a third descriptor, and modifying the transformer to reduce the distance.

[0014] In one embodiment, the modification step may include modifying the transformer to minimize the distance.

[0015] In one embodiment, the step of training a transformer using comparative learning may include the steps of comparing a second descriptor with a second descriptor using a discriminator, and determining whether a third descriptor is real or fake using a discriminator.

[0016] The descriptor generator may further include an inverse transformer, which may further include the steps of: using the inverse transformer to convert a third descriptor to a fourth descriptor; calculating the distance between the first descriptor and the fourth descriptor; and modifying the inverse transformer to reduce the distance.

[0017] In one embodiment, the first modality and the second modality may correspond to modalities used to record their respective data.

[0018] In one embodiment, the sensor may include one of a LiDAR sensor, a RADAR sensor, and a camera.

[0019] In one embodiment, the first encoder and the second encoder may each be an autoencoder.

[0020] In one embodiment, the autoencoder may be a variational autoencoder.

[0021] Another aspect of the present invention provides a computer implementation method for retrieving data of a first modality. This method includes the steps of: retrieving descriptors from a database of descriptors associated with data of a second modality; converting the retrieved descriptors into transformed descriptors using a reverse transformer, wherein the transformed descriptors are associated with the first modality and the first and second modalities are different; and retrieving data of the first modality associated with the transformed descriptors from a database.

[0022] The foregoing aspects may be expressed differently as a computer-implemented method for operating and moving an autonomous vehicle, the method comprising: obtaining real-time data from a sensor of a first modality; retrieving a descriptor from a database of descriptors associated with data of a second modality; using an inverse transformer to transform the retrieved descriptor into a transformed descriptor, the transformed descriptor being associated with the first modality, the first and second modalities being different; decoding the transformed descriptor of the first modality by a decoder to generate data of the first modality; comparing the real-time data with the generated data of the first modality; and operating and moving the autonomous vehicle based on the comparison.

[0023] In one embodiment, the first and second modalities may respectively correspond to the modalities of sensors used to record their respective data.

[0024] In one embodiment, the sensor may be one of a LiDAR sensor, a RADAR sensor, or a camera.

[0025] In one embodiment, the method may further comprise obtaining real-time data from a sensor of the first modality, comparing the real-time data with the retrieved data, and operating and moving the autonomous vehicle based on the comparison.

[0026] According to another aspect of the present invention, there is provided a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to execute the method according to any one of the preceding claims.

[0027] The subject matter of the present disclosure is best understood by reference to the accompanying drawings.

Brief Description of the Drawings

[0028] [Figure 1]Schematic diagram of AV according to one or more embodiments. [Figure 2] Block diagram of a variational autoencoder, one for each modality. [Figure 3] More detailed block diagram of the variational autoencoder of FIG. 2. [Figure 4] Block diagram of a classifier used to train an encoder from the variational autoencoder of FIG. 2 using paired data. [Figure 5] Block diagram of a distance comparator used to train an autoencoder from the variational autoencoder of FIG. 2 using paired data. [Figure 6] Block diagram representing the training of a decoder from the variational autoencoder of FIG. 2. [Figure 7] Block diagram for training a transformer to transform descriptors from the bottleneck of the variational autoencoder of FIG. 2 using unpaired data according to one or more embodiments, and training an inverse transformer to reverse the transformation of the descriptors. [Figure 8] Block diagram of the transformer of FIG. 7. [Figure 9] Block diagram of the inverse transformer of FIG. 7. [Figure 10] Block diagram for training a transformer to transform descriptors from the bottleneck of the variational autoencoder of FIG. 2 using paired data according to one or more embodiments, and training an inverse transformer to reverse the transformation of the descriptors. [Figure 11] [[ID=:30]]Block diagram for storing the transformed descriptors and updating the indices of the closest descriptors with the transformed descriptors according to one or more embodiments. [Figure 12] Block diagram representing the retrieval of data using the transformed descriptors according to one or more embodiments. [Figure 13] This is a flowchart of a computer implementation method for generating descriptors associated with data of the first modality. [Figure 14] This is a flowchart of a computer implementation method for training a descriptor generator to generate descriptors associated with data of the first modality. [Figure 15] This is a flowchart of the computer implementation method for retrieving data from the first modality. [Modes for carrying out the invention]

[0029] The embodiments described herein are embodied as instruction sets stored as electronic data on one or more storage media. Specifically, these instructions may be provided on a temporary or non-temporary computer-readable medium. When executed by a processor, the processor is configured to perform various methods described in the embodiments below. Thus, these methods may be computer implementation methods. Specifically, the processor and storage containing the instructions may be incorporated into a vehicle. The vehicle may be an AV (Aviation Vehicle).

[0030] The following embodiments provide specific exemplary examples, but these exemplary examples should not be construed as limiting, and the scope of protection is determined by the claims. Features of specific embodiments may be used in combination with features of other embodiments without extending the subject matter beyond the scope of this disclosure.

[0031] Referring to Figure 1, the AV10 may include a plurality of sensors 12. The sensors 12 may be mounted on the roof of the AV10. The sensors 12 may be communicatively connected to a computer 14. The computer 14 may be mounted on the AV10. The computer 14 may include a processor 16 and memory 18. The memory may include the non-temporary computer-readable medium described above. Alternatively, the non-temporary computer-readable medium may be located remotely and communicatively linked to the computer 14 via a cloud 20. The computer 14 may be communicatively linked to one or more actuators 22 to control one or more actuators 22 to move the AV10. The actuators may include, for example, a motor, a brake system, a power steering system, and the like.

[0032] Sensor 12 may include various sensor types. Examples of sensor types include LiDAR sensors, RADAR sensors, and cameras. Each sensor type may be called a sensor modality. Each sensor type may record data related to the sensor modality. For example, a LiDAR sensor may record LiDAR modality data.

[0033] The data may capture various scenes that the AV10 encounters. For example, the scenes may be visible scenes around the AV10, and may include roads, buildings, weather, and objects (such as other vehicles, pedestrians, animals, etc.).

[0034] Referring to Figure 2, an autoencoder 24 is trained for each modality. The autoencoder may be a variational autoencoder (VAE). The VAE 24 may be configured to receive data packets 26 of a specific modality, such as image data or LiDAR data. A scene may be captured in each data packet 26. The VAE 24 may include an encoder 28 and a decoder 30. The encoder may reduce the dimensions of the data packet 26 to a bottleneck distribution 32. This distribution may be called a descriptor 34. The decoder 30 may reconstruct the data packet 26 by increasing the dimensions of the descriptor 34 to the same size as the original data packet 26.

[0035] To train VAE24, the error between the reconstructed data packet and the original data packet is calculated. Backpropagation may be used to optimize the encoder and decoder hyperparameters.

[0036] The descriptor 34 may be stored in a database. The descriptor 34 may be used to regenerate a scene corresponding to the data packets associated with the descriptor 34. The regenerated scene may be used when the AV10 (Figure 1) operates the AV10. For example, the regenerated scene can be associated with a specific operating mode. Current sensor data can be compared with the regenerated scene, and the AV10 can be controlled to operate in the relevant operating mode based on how closely the current scene data matches the regenerated scene data. Alternatively, the regenerated scene data may be used in a simulator to expand the scenarios that the AV10 is trained to handle in real time.

[0037] Referring to Figure 3, VAE24 may receive data packets 26 from the sensor. Furthermore, VAE24 may receive synthesized data packets 26'. Synthesized data packets 26' may be data packets in which features from the original sensor data packets have been augmented, or data from simulation. For example, additional objects such as pedestrians or vehicles, or weather conditions such as rain or fog, may be added using a generative adversarial network. In this way, VAE24 may have a larger data train from which to train.

[0038] Typically, the VAE 24 may generate descriptors having a specific distribution, which may be a Gaussian distribution. However, in one or more embodiments, the distribution does not have to be a Gaussian distribution. Depending on the features in the scene of the data packet, the distribution may differ. For example, there may be a distribution where two dynamic objects exist in the scene, a specific distribution where a stationary vehicle exists in the scene, and a specific distribution where a specific weather condition such as fog exists. A lookup table called a codebook 36 may be provided. The codebook contains all distributions. A hidden layer may be extracted in the encoder 28, which contains features in the scene extracted from the original data packets 26, 26'. The features in the hidden layer may be used to identify one or more distributions from the lookup table. If multiple features exist, the multiple distributions may be statistically combined to obtain the distribution for a particular scene.

[0039] Since each VAE24 is trained independently, the descriptors are modality-specific to each modality of the VAE24. Thus, there is no correlation between descriptors. In other words, it is difficult to compare descriptors derived from data of different modalities. For example, it is possible to compare descriptors derived from image data with descriptors derived from LiDAR data. However, using cosine similarity or Euclidean distance, for example, matching scenes may have relatively short distances or no distance at all.

[0040] Referring to Figures 4 and 5, it is possible to use contrast learning to reduce inconsistencies between descriptors of different modalities.

[0041] Referring to Figure 4, controlled learning may be performed by training each encoder 28 using the classifier 38 as a supervisor. For example, a scene 26A of a first modality, such as an image, may be input to the first encoder 28A. A scene 26B of a second modality, such as LiDAR data, may be input to the second encoder 28B. The scene of the second modality may also be input to the third encoder 28C. The inputs to the second and third encoders 28A and 28B may differ in that the input to the second encoder may positively match the scene of the first modality, while the scene input to the third encoder may not match or may negatively match the scene of the first modality. Since the data of the first and second modalities are paired, positive and negative matches are known. "Paired" means that they represent the same scene and may be temporally synchronized when captured.

[0042] The classifier 38 is used to train the parameters of the first and second encoders 28A and 28B, respectively. Training means that the first and second descriptors 34A and 34B, derived from the first and second encoders 28A and 28B, respectively, categorize the scenes in the same way. Furthermore, the third descriptor 34C, derived from the third encoder 28C, categorizes the scenes differently.

[0043] Referring to Figure 5, comparative learning may be performed using the first and second distance matchers 40 and 42.

[0044] The first distance matcher 40 may calculate the distance between the first and second descriptors 34A, 34B. The first and second encoders 28A, 28B are trained to derive the first and second descriptors 34A, 34B having a distance of zero or at least very small or negligible. The first distance matcher 40 may calculate the distance using Euclidean distance, cosine similarity, a trained matcher (e.g., a neural network), or other distance metrics.

[0045] The second distance matcher 42 may calculate the distance between the second and third descriptors 34B, 34C. The second and third encoders 28B, 28C are trained to derive the second and third descriptors 34B, 34C having very large distances, which may tend to be infinite. The second distance matcher 42 may calculate the distance using Euclidean distance, cosine similarity, a trained matcher (e.g., a neural network), or other distance metrics.

[0046] Figure 6 shows a block diagram illustrating how each decoder is trained, in addition to training the encoder using the comparative learning method shown in Figure 5.

[0047] For example, there are first to third decoders 30A to 30C to be trained. Each decoder 30A to 30C corresponds to one of the first to third encoders 28A to 28C, respectively. Each decoder is trained by comparing a replayed scene of a given modality with the original scene 26 of that modality and modifying the hyperparameters of the respective decoder to reduce or minimize the error.

[0048] The algorithm described with reference to Figures 4-6 works well when the data is paired. However, when the data is not paired, for example, when it is from a single modality, it may not be possible to make the descriptors obtained from different modalities comparable.

[0049] To mitigate this problem, an embodiment of the descriptor generator 45 shown in Figure 7 is provided.

[0050] Referring to Figure 7, the descriptor generator 45 includes a first encoder 28A, a second encoder 28B, a transformer 46, a discriminator 48, an inverse transformer 50, and a distance matcher 52. The first training data 26A (e.g., sample scenes) and the second training data 26B (e.g., sample scenes) for the first and second modalities 26A and 26B, respectively, are input to the first and second encoders 28A and 28B. The first and second encoders 28A and 28B are trained as described above with reference to Figures 2 and 3. In other words, the first and second encoders 28A and 28B are trained independently. The first encoder 28A uses the first training data of the first modality to generate a first descriptor 34_1 corresponding to the first data 26A of the first modality. The second encoder 28B uses the second training data of the second modality to generate a second descriptor 34_2 corresponding to the second data 26B of the second modality. The first and second modalities may differ in that they are acquired by different sensor types. For example, the first modality may be LiDAR sensor data and the second modality may be camera image data.

[0051] The transformer 46 may be trained to convert the first descriptor 34_1 to the third descriptor 34_3. The third descriptor may correspond to the second modality. In other words, the transformer 46 is configured to convert the descriptor from the first modality to the second modality so that the third descriptor 34_3 appears to have been generated by the first encoder 28A rather than the second encoder 28B.

[0052] Referring briefly to Figure 8, the transformer 46 includes a decoder 53 and an encoder 54. The decoder 53 is configured to generate third data for a first modality, such as a camera image. The encoder 54 is configured to generate a third descriptor 34_3, which corresponds to a second modality.

[0053] Referring further to Figure 7, the descriptor generator 45 compares the second descriptor 34_2 and the third descriptor 34_3 using a discriminator 48. The discriminator 48 is configured to determine whether the third descriptor 34_3 is genuine or fake. For example, a genuine determination means that the transformer 46 has successfully generated a third descriptor 34_3 that the discriminator 48 believes is directly derived from the data sample 26B of the second modality and not from a transformed form of the data sample 26A of the first modality. A fake determination means that the transformer 46 has generated a third descriptor 34_3 that the discriminator 48 does not believe is directly derived from the data 26B of the second modality. The discriminator 48 may be a neural network.

[0054] If the discriminator 48 determines that the third descriptor 34_3 is fake, the transformer 46 is modified. Specifically, the parameters of the transformer 46 are modified until it is determined to be a genuine descriptor.

[0055] The inverse transformer 50 may also transform the third descriptor 34_3 into the fourth descriptor 34_4.

[0056] Referring briefly to Figure 9, the inverse transformer 50 may include a decoder 56 and an encoder 58. The decoder 56 may generate data such as LiDAR data of a first modality using a third descriptor corresponding to a second modality, such as a camera image. The encoder 58 may generate a fourth descriptor 34_4 from the generated data of the second modality.

[0057] Referring further to Figure 7, the distance matcher 52 is configured to calculate the distance between the first descriptor and the fourth descriptor. The distance matcher 52 may perform distance matching using any of the following: Euclidean distance, cosine similarity, a trained matcher (e.g., a neural network), or any other suitable distance metric. The distance may be interpreted as an error value. The inverse transformer 50 may be modified to shorten or minimize the distance. In certain circumstances, the transformer 46 may also be modified to shorten or minimize the distance.

[0058] One reason for using the distance matcher 52 is to avoid a situation where the transformer 46 generates a third descriptor 34_3 that is considered authentic but has different characteristics from the original data 26A. For example, the original image may contain three pedestrians and four moving vehicles. If the third descriptor 34_3 generates an image that contains no pedestrians but two animals and a stationary car, the third descriptor 34_3 is considered authentic in that it is derived from the second modality.

[0059] In this way, data from the first modality can be indexed using descriptors of target modalities, such as a second modality that is different from the first modality. Thus, all modalities may be indexed using descriptors converted to target modalities. As a result, data does not need to be paired in order to be indexed and compared with data from similar or identical scenes, even if derived from different sensor types.

[0060] Referring to Figure 10, one embodiment of the descriptor generator 145 is provided. The descriptor generator 145 is the same as the descriptor generator in Figure 7, except that the discriminator 48 is replaced by a distance matcher 160. To avoid duplication, only the differences between the embodiment in Figure 7 and the embodiment in Figure 10 will be described, and similar features will be preceded by 1.

[0061] The third descriptor 134_3 is compared to the second descriptor 134_2 by the distance matcher 160. The distance matcher 160 may calculate the distance using Euclidean distance, cosine similarity, a trained matcher (e.g., a neural network), or any appropriate matching metric.

[0062] The transformer 146 may be modified to shorten or minimize the distance based on the distance between the second and third descriptors 134_2 and 134_3. For example, the parameters of the transformer 146 may be modified.

[0063] Referring to Figure 11, the descriptor generator 45 is shown as generating descriptors for indexing data of the first modality.

[0064] The first data 26A associated with the first modality (e.g., LIDAR sensor data) may be used as input to the first encoder 28A. Similar to the embodiments described above, the first encoder 28A generates a first descriptor 34_1. The transformer 46 converts the first descriptor to a third descriptor 34_3, the third descriptor corresponding to the second modality. The third descriptor 34_3 is stored in the database 62. The database 62 also contains other descriptors, all of which correspond to the second modality and are derived either directly from sensor data of the second modality or by conversion from other modalities to the second modality.

[0065] The distance matcher 64 directly receives a third descriptor. The third descriptor 34_3 is compared by the distance matcher 64 to a descriptor in the database 62. The comparison may be performed using methods such as Euclidean distance, cosine similarity, or a trained matcher (macher) such as a neural network. The result of the distance matcher 64 is input to index 66 of the nearest descriptor.

[0066] Referring to Figure 12, the requesting system 68 may request a scene in the first modality. The requesting system may identify the scene using, for example, the index shown in Figure 11. The requesting system 68 may retrieve a third descriptor 34_3 from the descriptor database 62. The third descriptor 34_3 is associated with data in the second modality, for example, camera image data.

[0067] The inverse transformer 50 may convert the third descriptor 34_3 to the fourth descriptor 34_4 of the first modality. The fourth descriptor 34_4 may be called the converted descriptor. The converted descriptor 34_4 may be of the first modality. The fourth descriptor 34_4 may be input to the decoder 56, which may be the decoder shown in Figure 9. The decoder 56 may generate a reconstructed image 70A of the first modality (e.g., LiDAR data).

[0068] During execution, the computer 14 (Figure 1) may acquire real-time data from the sensor 12 of the first modality (for example, a LiDAR sensor). The computer 14 may compare the real-time data with the retrieved data. Thus, the computer 14 may also be the requesting system 68. The computer 14 may operate and move the AV 10 based on the comparison. For example, since the AV 10 may have previously passed through a very similar scene, it may use a previously used trajectory to construct a trajectory for moving through the current scene.

[0069] While the above description provides details regarding embodiments, the subject matter of this disclosure may be summarized with reference to Figures 13-15.

[0070] Referring to Figure 13, a computer implementation method for generating descriptors associated with data of a first modality is provided. This method includes: receiving (step S100) first data associated with a first modality and second data associated with a second modality, wherein the first and second modalities are different; generating (step S102) generating first and second descriptors corresponding to the first and second modalities by encoding the first and second data using the first and second encoders, respectively, wherein the first and second encoders are trained on the first training data of the first modality and the second training data of the second modality, respectively; converting (step S104) the first descriptors into third descriptors, wherein the third descriptors correspond to the second modality; and storing the third descriptors in a database (step S106).

[0071] Referring to Figure 14, a computer implementation method is provided for training a descriptor generator to generate descriptors for indexing data of a first modality, the descriptor generator comprising a first encoder, a second encoder, and a transformer. The method includes the steps of: training a first encoder to generate a first descriptor corresponding to first data of a first modality using first training data of a first modality (step S200); training a second encoder to generate a second descriptor corresponding to second data of a second modality using second training data of a second modality, wherein the first and second modalities are different (step S202); and training a transformer to transform the first descriptor into a third descriptor corresponding to a second modality using comparative learning based on the first and second training data (step S204).

[0072] Referring to Figure 15, a computer implementation method for retrieving data for a first modality is provided. This method includes the steps of: retrieving descriptors from a database of descriptors associated with data for a second modality (step S300); converting the retrieved descriptors into converted descriptors using an inverse transformer, wherein the converted descriptors are associated with the first modality, and the first and second modalities are different (step S302); and retrieving data for the first modality associated with the converted descriptors from the database (step S304).

[0073] The embodiments described above are provided to illustrate the subject matter of this disclosure, but the features of the embodiments should not be construed as limiting the scope of protection. To avoid any doubt, the scope of protection is defined by the following claims.

Claims

1. A computer implementation method for generating descriptors associated with data of a first modality, A step of receiving first data associated with a first modality and second data associated with a second modality, wherein the first and second modalities are different, A generating step of generating first and second descriptors corresponding to each first and second modality by encoding the first and second data using the respective first and second encoders, wherein the first and second encoders are trained on the first training data of the first modality and the second training data of the second modality, respectively. A step of converting the first descriptor into a third descriptor, wherein the third descriptor corresponds to the second modality, The steps include storing the third descriptor in the database, Computer implementation methods, including those mentioned above.

2. The aforementioned conversion step is: A decoding step comprising: decoding the first descriptor into decoded third data corresponding to the second modality using a decoder, wherein the decoder is trained to generate data for the second modality; The steps include encoding the third data to generate the third descriptor corresponding to the second modality using a third encoder, The computer implementation method according to claim 1, including the method described in claim 1.

3. The computer implementation method according to claim 1, wherein the first modality and the second modality correspond to the modalities of sensors used to record their respective data.

4. The computer implementation method according to claim 3, wherein the sensor comprises one of a LiDAR sensor, a RADAR sensor, and a camera.

5. The computer implementation method according to claim 1, wherein the first encoder and the second encoder are from their respective autoencoders, and the respective first descriptor and second descriptor are representations corresponding to the bottleneck of their respective autoencoders.

6. The computer implementation method according to claim 5, wherein the autoencoder is a variational autoencoder.

7. A computer implementation method for training a descriptor generator for generating descriptors associated with data of a first modality, wherein the descriptor generator comprises a first encoder, a second encoder, and a transformer, and the method is The steps include training the first encoder to generate a first descriptor corresponding to the first data of the first modality using first training data of the first modality, A step of training the second encoder to generate a second descriptor corresponding to the second data of the second modality using second training data of the second modality, wherein the first and second modalities are different, The steps include training the transformer to convert the first descriptor to a third descriptor corresponding to the second modality using controlled learning based on the first and second training data, Computer implementation methods, including those mentioned above.

8. The step of training the transformer using comparative learning is: A step of determining the distance between the second descriptor and the third descriptor, A step of modifying the transformer to shorten the aforementioned distance, The computer implementation method according to claim 7, including the method described in claim 7.

9. The step of training the transformer using comparative learning is: A step of comparing the second descriptor and the third descriptor using a discriminator, The steps include determining whether the third descriptor is genuine or fake using the discriminator, The computer implementation method according to claim 7, including the method described in claim 7.

10. The descriptor generator further comprises an inverse transformer, and the method is The steps include converting the third descriptor to a fourth descriptor using the inverse transformer, A step of calculating the distance between the first descriptor and the fourth descriptor, A step of modifying the inverse transformer to shorten the aforementioned distance, The computer implementation method according to claim 7, further comprising:

11. The computer implementation method according to claim 7, wherein the first modality and the second modality correspond to the modality of a sensor used to record the respective data.

12. The computer implementation method according to claim 11, wherein the sensor comprises one of a LiDAR sensor, a RADAR sensor, and a camera.

13. The computer implementation method according to claim 7, wherein the first encoder and the second encoder are each encoders of an autoencoder.

14. The computer implementation method according to claim 13, wherein the autoencoder is a variational autoencoder.

15. A computer implementation method for extracting data from the first modality, The steps include retrieving descriptors from a database of descriptors associated with data of the second modality, A step of using an inverse transformer to convert the extracted descriptor into a transformed descriptor, wherein the transformed descriptor is associated with the first modality, and the first and second modalities are different. The steps include: retrieving data for the first modality associated with the converted descriptor from the database; Computer implementation methods, including those mentioned above.

16. The computer implementation method according to claim 15, wherein the first and second modalities correspond to the modalities of sensors used to record their respective data.

17. The computer mounting method according to claim 16, wherein the sensor is one of a LiDAR sensor, a RADAR sensor, or a camera.

18. The steps include acquiring real-time data from the sensor of the first modality, The steps include comparing the real-time data with the extracted data, The steps include operating and moving the autonomous vehicle based on the above comparison, The computer implementation method according to claim 15, further comprising:

19. A temporary or non-temporary computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

Images