Model pre-training optimization method and system for fusing multi-modal data

Abstract

The application provides a model pre-training optimization method and system for fusing multi-modal data, and relates to the technical field of multi-modal technology. The method comprises the following steps: after obtaining voice modal data and other modal data, sampling at low resolution and high resolution two scales respectively, generating corresponding voice vector sets and calculating uncertainty metrics; globally aligning the voice and other modal data at the first resolution scale, and transferring the global alignment result to the second resolution scale to perform refinement correction, and finally returning to the first resolution scale, thereby forming a multiple mesh format back-and-forth iteration; by monitoring and observing noise and state noise in real time and integrating them into the uncertainty metric, the weighted coefficient can be dynamically reduced during training for high-noise sections, and a higher weight is given to low-noise or stable sections to strengthen effective features; the application can adaptively suppress noise interference and retain voice mutation details in a multi-modal scene, and has better robustness and generalization performance.

Images