A multimodal target detection method and system suitable for missing modality

Abstract

The invention discloses a multi-modal target detection method and system suitable for missing modalities. The method includes the following steps: S1. Train a neural network unit with multi-modal data in a data set; The modal data is input into the trained neural network unit for detection, and the detection results are saved; S2, the modal data of other dimensions are extracted; the pseudo-modal feature tensor is generated by the generation network unit, and then spliced ​​and input into the attention network unit, information fusion unit, and discriminating network unit, until the training of the generating network unit is completed; S3, collecting modal data in real time through data acquisition equipment, and using the trained neural network unit to generate target types and identifiers. On the premise of avoiding noise introduction and loss of feature information, the method of the invention generates the missing modal data virtually, which greatly reduces the calculation amount and complexity of the model, and can improve the representation consistency of the generated modalities.

Description

technical field [0001] The invention relates to the technical field of automatic driving, in particular to a multimodal target detection method and system suitable for missing modalities. Background technique [0002] With the gradual rise and progress of various technologies such as Computer Vision (CV) and machine learning in the field of artificial intelligence, cameras and image processing equipment can replace the human eye and detect, identify, track, and count etc. Among them, target detection is the basic task of target recognition, tracking and counting. Especially in the field of autonomous driving, in-vehicle terminals usually need to combine data acquisition devices such as lidar, cameras, and roadside edge equipment to collect modal data of multiple different dimensions, and perform target detection according to the collected modal data of different dimensions. recognition in order to improve the accuracy of target detection and recognition. [0003] However,...

AI Technical Summary

Problems solved by technology

[0003] However, in the actual application process, it is inevitable that modal data will be lost, such as: camera failure, encountering extreme weather conditions, etc., may cause modal data loss or partial information loss

[0004] Aiming at the problem of missing modal data and partial information loss in complex environments, the solutions of existing technologies are usually: 1. Delete samples with missing modals, use data completion technology to fill missing modal features, and then use the existing However, these operations will lead to problems such as the introduction of additional noise in the detection results; 2. Use matrix decomposition techniques to obtain consistent representations of missing modalities or use Laplacian regularization to complement missing modal similarities Although it effectively avoids the introduction of additional noise, it causes the loss of effective characteristic information of the mode

Images