An open vocabulary semantic segmentation method and device, electronic equipment and storage medium

Abstract

The application relates to the field of artificial intelligence and computer vision, in particular to an open vocabulary semantic segmentation method and device, electronic equipment and storage medium, the application constructs an automatic process of "semantic guidance-iterative optimization-segmentation verification-closed loop feedback", and fuses the open vocabulary semantic understanding ability of a multi-modal large language model with the general segmentation ability of a visual basic segmentation model with general pixel-level segmentation ability. Without target category label data, only according to the semantic target name provided by the user, the accurate semantic segmentation of any category target in the image can be realized. The segmentation result is endowed with semantics by the large language model, and the geometric accuracy is guaranteed by the basic segmentation model. The method overcomes the limitation that the traditional method is difficult to handle new classes in the open world, improves the semantic accuracy and geometric integrity of the segmentation result, and provides an efficient and low-cost solution for industrial detection, robot vision and other open scenarios.

Images