A method, device and apparatus for generating a depth map based on multi-view stereo vision

By segmenting and semantic feature extraction of multi-view images, combined with the context information of the visual language model, the matching accuracy problem of traditional methods under lighting changes and occlusion is solved, and a higher multi-view stereoscopic visual matching accuracy is achieved.

CN118864692BActive Publication Date: 2024-11-29NAT UNIV OF DEFENSE TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411344680.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-25
Publication Date
2024-11-29
Estimated Expiration
2044-09-25

AI Technical Summary

Technical Problem

Traditional multi-view stereo vision technology is difficult to ensure the accuracy of cross-view matching under changes in lighting conditions and occlusion, resulting in a reduction in pixel-level matching accuracy. The existing improvement methods still have limitations.

Method used

By segmenting multi-view images, semantic features and appearance features are obtained, and multi-level semantic features are aggregated, combining the context semantic features of visual language big models, a more accurate depth map is generated.

Benefits of technology

Improve the accuracy of pixel-level matching across the visual image, reduce matching errors caused by lighting changes and occlusion, and enhance the system's robustness to complex scenes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118864692B_ABST
    Figure CN118864692B_ABST
Patent Text Reader

Abstract

The present invention relates to a method, device and equipment for generating a depth map based on multi-view stereo vision. The method comprises: performing image segmentation on a set of input multi-view images to obtain a plurality of image segments and the confidence corresponding to each image segment. The semantic features of the retained image segments and the contextual semantic features of the multi-view images are obtained through a pre-trained large visual language model, and the contextual semantic features are assigned to each pixel in each image segment. According to the semantic features and contextual semantic features of the retained image segments to which each pixel belongs, multi-level semantic feature aggregation is performed to obtain the semantic aggregation features of each pixel. The target aggregation features are aggregated with the appearance features to obtain the target aggregation features. The depth map of the reference image in the multi-view image is generated according to the target aggregation features. The method can improve the accuracy of multimodal feature aggregation in large indoor scenes with cluttered, changing lighting conditions and open vocabulary objects.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer stereo vision technology, and in particular to a method, device and equipment for generating a depth map based on multi-view stereo vision. Background Art

[0002] The core of Multi-View Stereo (MVS) technology is to reconstruct 3D scenes by establishing pixel-level correspondences across different views. Traditional MVS methods usually rely on learning-based appearance features extracted from RGB images to find matching points between images from different perspectives. These appearance features usually contain information such as color and texture. After being extracted and aggregated by deep learning models, preliminary pixel-level correspondences can be established between different views. However, since these methods mainly rely on visual features in RGB images, they are prone to serious problems when lighting conditions change and occlusion occurs.

[0003] Variations in lighting conditions are a major challenge in multi-view stereo. In real scenes, lighting is often dynamic and inconsistent, which results in significant differences in the appearance features of the same object in different views. For example, in a set of multi-view images, an object may appear very bright in one image due to direct sunlight, but appear dim in another image due to shadows. This lighting inconsistency makes it difficult to ensure the accuracy of cross-view matching by aggregating appearance features based on RGB images, thereby reducing the accuracy of pixel-level matching.

[0004] In addition, occlusion also poses a great challenge to the pixel-level matching of MVS. In complex scenes, occlusion between objects is common. For example, furniture, walls or other objects may partially block the target object in the field of view. These occlusions cause the target object in certain viewing angles to be completely invisible in the image. Traditional feature extraction methods based on RGB images often cannot provide reliable matching information in these cases, further affecting the accuracy of stereo vision reconstruction.

[0005] In order to solve these problems, existing technologies have introduced some improved methods. For example, combining depth maps, normal maps or using multi-scale feature extraction technology to increase adaptability to complex scenes. However, these methods are still limited in practical applications and cannot completely overcome the problems caused by illumination changes and occlusions. Even after introducing multimodal auxiliary information, such as depth maps or normal maps, the system still faces the problem of feature inconsistency caused by changes in lighting conditions and occlusions. This inconsistency will introduce noise in the feature aggregation process, resulting in a significant decrease in the final matching accuracy.

[0006] Therefore, although the existing technology has made many optimizations in multimodal feature matching, it still has significant limitations when dealing with changes in lighting conditions and occlusion of invisible objects. Faced with these challenges, how to further improve the robustness of multimodal feature matching has become a key technical problem in achieving high-precision 3D reconstruction. Summary of the invention

[0007] Based on this, it is necessary to provide a method, device and equipment for generating a depth map based on multi-view stereo vision, which can improve and promote the accuracy of pixel-level matching across views, in order to address the above technical problems.

[0008] A method for generating a depth map based on multi-view stereo vision, the method comprising:

[0009] Image segmentation is performed on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment.

[0010] The semantic features of the retained image fragments and the contextual semantic features of the multi-view images are obtained through a pre-trained visual language model, and the contextual semantic features are assigned to each pixel in each image fragment.

[0011] According to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, multi-level semantic feature aggregation is performed to obtain the semantic aggregation features of each pixel.

[0012] The semantic aggregation features and appearance features of each pixel are aggregated to obtain the target aggregation features corresponding to each pixel.

[0013] The depth map of the reference image in the multi-view image is generated based on the target aggregated features of each pixel.

[0014] A depth map generation device based on multi-view stereo vision, the device comprising:

[0015] The image segmentation module is used to perform image segmentation on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment.

[0016] The semantic feature extraction module is used to obtain the semantic features of the retained image fragments and the contextual semantic features of the multi-view images through a pre-trained visual language model, and assign the contextual semantic features to each pixel in each image fragment.

[0017] The semantic feature aggregation module is used to perform multi-level semantic feature aggregation according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features to obtain the semantic aggregation features of each pixel.

[0018] The semantic-appearance feature aggregation module is used to aggregate the semantic aggregation features and appearance features of each pixel to obtain the target aggregation features corresponding to each pixel.

[0019] The depth map aggregation module generates the depth map of the reference image in the multi-view image based on the target aggregation features of each pixel.

[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Image segmentation is performed on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment.

[0022] The semantic features of the retained image fragments and the contextual semantic features of the multi-view images are obtained through a pre-trained visual language model, and the contextual semantic features are assigned to each pixel in each image fragment.

[0023] According to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, multi-level semantic feature aggregation is performed to obtain the semantic aggregation features of each pixel.

[0024] The semantic aggregation features and appearance features of each pixel are aggregated to obtain the target aggregation features corresponding to each pixel.

[0025] The depth map of the reference image in the multi-view image is generated based on the target aggregated features of each pixel.

[0026] The above-mentioned depth map generation method, device and equipment based on multi-view stereo vision, first, in the multi-view image segmentation stage, by segmenting the input multi-view image, a number of image segments are obtained, and the confidence of each segment is calculated. This process ensures that when processing the image, the key information at different perspectives can be identified, thereby laying the foundation for the subsequent semantic feature extraction. Next, the pre-trained visual language large model is used to obtain the semantic features of these image segments, and the contextual semantic features of the entire multi-view image are extracted at the same time, and these contextual semantic features are assigned to each pixel of each image segment. The core of this step is that by introducing the visual language large model, the system can understand and associate the semantic information in complex scenes, not only relying on visual features, but also combining language and contextual information. It is particularly effective when dealing with illumination changes and invisible objects. Illumination changes and occlusions usually cause errors in pixel-level matching of traditional visual methods. By introducing contextual semantic features, the system can more accurately understand the elements in the scene and their relative relationships, reducing matching errors caused by inconsistent illumination or occlusion. Subsequently, multi-level semantic feature aggregation is performed based on the semantic features of the retained image fragment to which each pixel belongs and the contextual semantic features. This step can combine local information at the pixel level with global semantic information through hierarchical feature aggregation to generate more representative semantic aggregated features. This multi-level feature aggregation method helps to distinguish the contours of objects and occluded areas under the influence of lighting in complex scenes, so that semantic features and visual features are more closely combined. Finally, the semantic aggregated features and appearance features of each pixel are further feature aggregated to obtain the target aggregated features corresponding to each pixel. The effective integration of semantic information and visual information makes it possible to more accurately reflect the depth relationship in the scene when generating the depth map of the reference image, thereby greatly improving the accuracy of multi-view image matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a VL-MVSNet framework based on learning in one embodiment;

[0028] Figure 2 is a schematic diagram of a flow chart of a method for generating a depth map based on multi-view stereo vision in one embodiment;

[0029] Figure 3 A schematic diagram of a channel conversion network framework in an embodiment;

[0030] Figure 4 is a structural block diagram of a depth map generating device based on multi-view stereo vision in one embodiment;

[0031] Figure 5FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] The depth map generation method based on multi-view stereo vision provided by the present invention can be applied to Figure 1 In the learning-based VL-MVSNet framework shown in the figure, given multiple overlapping RGB images, VL-MVSNet first performs segmentation to generate multi-level segments and corresponding confidence maps, and then extracts multi-level CLIP features for each segment through the contrastive visual language model (CLIP). These features, as well as the semantic features derived from the confidence-driven feature aggregation module for each pixel and the appearance features extracted by the 2D U-Net, are then input into the channel conversion network (CTN). The network generates multimodal features, which are ultimately input into the multi-view stereo network (MVSNet) or other multi-view stereo (MVS) frameworks to generate a depth map of the reference image.

[0034] In one embodiment, Figure 2 As shown in FIG. 1 , a depth map generation method based on multi-view stereo vision is provided, and the method is applied to Figure 1 The VL-MVSNet framework in is used as an example to illustrate, including the following steps:

[0035] Step 202: perform image segmentation on a set of input multi-view images to obtain a number of image segments and a confidence level corresponding to each image segment.

[0036] Step 204 , using a pre-trained visual language model, obtains semantic features of the retained image segments and contextual semantic features of the multi-view images, and assigns the contextual semantic features to each pixel in each image segment.

[0037] Step 206 , performing multi-level semantic feature aggregation according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, to obtain the semantic aggregation features of each pixel.

[0038] Step 208 , perform feature aggregation on the semantic aggregation features and appearance features of each pixel to obtain the target aggregation features corresponding to each pixel.

[0039] Step 210 : generating a depth map of a reference image in a multi-view image according to the target aggregation feature of each pixel.

[0040] In the above-mentioned depth map generation method based on multi-view stereo vision, first, in the multi-view image segmentation stage, the input multi-view image is segmented to obtain several image segments, and the confidence of each segment is calculated. This process ensures that when processing the image, the key information at different perspectives can be identified, thereby laying the foundation for the subsequent semantic feature extraction. Next, the pre-trained visual language large model is used to obtain the semantic features of these image segments, and the contextual semantic features of the entire multi-view image are extracted at the same time, and these contextual semantic features are assigned to each pixel of each image segment. The core of this step is that by introducing the visual language large model, the system can understand and associate the semantic information in complex scenes, not only relying on visual features, but also combining language and contextual information. It is particularly effective when dealing with illumination changes and invisible objects. Illumination changes and occlusions usually cause errors in pixel-level matching of traditional visual methods. By introducing contextual semantic features, the system can more accurately understand the elements in the scene and their relative relationships, reducing matching errors caused by inconsistent illumination or occlusion. Subsequently, multi-level semantic feature aggregation is performed based on the semantic features of the retained image fragment to which each pixel belongs and the contextual semantic features. This step can combine local information at the pixel level with global semantic information through hierarchical feature aggregation to generate more representative semantic aggregated features. This multi-level feature aggregation method helps to distinguish the contours of objects and occluded areas under the influence of lighting in complex scenes, so that semantic features and visual features are more closely combined. Finally, the semantic aggregated features and appearance features of each pixel are further feature aggregated to obtain the target aggregated features corresponding to each pixel. The effective integration of semantic information and visual information makes it possible to more accurately reflect the depth relationship in the scene when generating the depth map of the reference image, thereby greatly improving the accuracy of multi-view image matching.

[0041] In one of the embodiments, a fast segmentation arbitrary model is used to segment a set of input multi-view images according to a preset RGB image source to generate a plurality of multi-level image segments and a confidence level corresponding to each image segment.

[0042] It is worth noting that the Fast Segmentation Arbitrary Model (FastSAM) is used, which is a lightweight CNN-based solution for segmenting arbitrary tasks that reduces computational requirements while maintaining competitive performance. FastSAM consists of two sequential modules: full instance segmentation and prompt-guided selection. This solution only uses the full instance segmentation module, which uses YOLOv8-seg to generate segmentation masks for all instances in the image. During FastSAM inference, multi-scale instances (e.g., objects, fragments) are generated. This is achieved by using a smaller confidence threshold for object detection, allowing more fragments to be involved.

[0043] In one of the embodiments, a pre-trained visual language model is used to extract semantic features of the top three retained image segments corresponding to the maximum confidence values ​​in each level of image segment set, generate contextual semantic features of the multi-view image, and assign the contextual semantic features to each pixel in each image segment.

[0044] It is worth noting that this is achieved by feeding the cropped image of the segmented instance into the CLIP model and assigning CLIP features to each pixel of the cropped image. Therefore, the output pixel-by-pixel features will contain the multi-level semantics of the scene. For each pixel, this scheme retains CLIP features extracted from up to 3 crops with the highest detection confidence in FastSAM segmentation. In addition, the unsegmented image is fed into CLIP to generate contextual features for the entire scene, which are assigned to each point of the input image, showing a good balance between accuracy and computational cost in scenes with multiple objects.

[0045] In one embodiment, according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, a multi-level semantic feature aggregation is performed through a fast segmentation arbitrary model to obtain the semantic aggregation feature of each pixel:

[0046] ;

[0047] in, The semantic aggregation feature for each pixel, To quickly segment any pair of models, For fast segmentation, any model detects confidence in each retained image segment. is the global information of all image segments, For each preserved image segment, is element-wise multiplication, is a multi-layer perceptron.

[0048] It is worth noting that, assuming is the input image, are the fragments detected by FastSAM. is the number of detected fragments. yes CLIP features. The features of the entire scene are Specifically, Includes global information indicating the type and layout of the entire scene, and the rest includes multi-scale semantics of specific objects in the image. For each pixel, select the CLIP features extracted from up to 3 segments that contain the pixel with the highest detection confidence. Denote them as and Note that for pixels belonging to less than 3 segments, the redundant features are set to vector 0. Multi-level CLIP feature aggregation is performed under the guidance of the detection confidence in Fast-SAM.

[0049] In one embodiment, a channel conversion network is constructed. The channel conversion network includes three cross-attention layers:

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] in, is the cross attention vector, is the query vector, is the key vector, is a value vector, They are the learning weights of the query vector, the learning weights of the key vector, and the learning weights of the value vector.

[0055] It is worth noting that if Figure 3 As shown in Figure 1, the framework of the channel conversion network (CTN) contains three cross-attention layers, each followed by two AddNorm layers and a feed-forward layer. If the channel in the BN layer has a small scaling factor, the corresponding channels in the feature map before the BN layer will be selected from all sub-networks (as shown by the dotted lines in the figure), and then the features in these channels will be input into the CTN to obtain new features. Given the i-th input data from M (M=2) modalities, it is represented as Represented as input modality. Where C represents the number of channels, H and W represent the height and width of the feature map. Indicates The l-th layer feature map of the sub-network, and The BN layer performs Normalization, and then affine transformation, that is:

[0056] ;

[0057] in and Calculate the mean and standard deviation of all activations at all pixel locations (H and W) of the current mini-batch, respectively; and are the trainable offset and scaling factor respectively; is a small constant to avoid division by zero. The (l+1)th layer will As the input of the nonlinear function. Evaluated the input during training and output If , then relative to The loss gradient will approach 0, which means will lose their influence on the final prediction and thus become redundant. Therefore, we pick out the channels with small scaling factors and the corresponding channels of other sub-networks, input them into CTN, and then replace these channels with the features in CTN.

[0058] In one embodiment, the semantic aggregation features of each pixel and the appearance features of the multi-view image are input into the channel conversion network for feature aggregation to obtain the target aggregation features corresponding to each pixel:

[0059] ;

[0060] in, is the target aggregation feature, is the l-th layer feature map on the c-th channel of the m-th subnetwork, is the average of all activations at all pixel locations in the current image segment, is the scaling factor of the channel conversion subnetwork, is the offset of the channel conversion subnetwork, is the standard deviation of all activations at all pixel locations in the current image segment, is a constant.

[0061] In one of the embodiments, a 3D cost volume is constructed based on the target aggregation features of each pixel, and after training the channel conversion network using a supervised learning method, an optimized target aggregation feature is output, and the optimized target aggregation feature generates a depth map of a reference image in a multi-view image through a 3D point cloud.

[0062] It is worth noting that since the visual language model contains knowledge of open vocabulary objects, the extracted semantic features can enhance pixel-level correspondences, and they support depth map or 3D point cloud queries given an open text query category, which is beneficial to various downstream applications.

[0063] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0064] In one embodiment, Figure 4 As shown, a depth map generation device based on multi-view stereo vision is provided, including: an image segmentation module 402, a semantic feature extraction module 404, a semantic feature aggregation module 406, a semantic-appearance feature aggregation module 408 and a depth map aggregation module 410, wherein:

[0065] The image segmentation module 402 is used to perform image segmentation on a set of input multi-view images to obtain a number of image segments and a confidence level corresponding to each image segment.

[0066] The semantic feature extraction module 404 is used to obtain the semantic features of the retained image segments and the contextual semantic features of the multi-view images through a pre-trained visual language model, and assign the contextual semantic features to each pixel in each image segment.

[0067] The semantic feature aggregation module 406 is used to perform multi-level semantic feature aggregation according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, so as to obtain the semantic aggregation features of each pixel.

[0068] The semantic-appearance feature aggregation module 408 is used to aggregate the semantic aggregation features and appearance features of each pixel point to obtain the target aggregation features corresponding to each pixel point.

[0069] The depth map aggregation module 410 generates a depth map of the reference image in the multi-view image according to the target aggregation feature of each pixel.

[0070] For the specific definition of a depth map generation device based on multi-view stereo vision, please refer to the definition of a depth map generation method based on multi-view stereo vision above, which will not be repeated here. Each module in the above-mentioned depth map generation device based on multi-view stereo vision can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0071] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a depth map generation method based on multi-view stereo vision is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0072] Those skilled in the art will understand that Figure 4-Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0073] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0074] Image segmentation is performed on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment.

[0075] The semantic features of the retained image fragments and the contextual semantic features of the multi-view images are obtained through a pre-trained visual language model, and the contextual semantic features are assigned to each pixel in each image fragment.

[0076] According to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, multi-level semantic feature aggregation is performed to obtain the semantic aggregation features of each pixel.

[0077] The semantic aggregation features and appearance features of each pixel are aggregated to obtain the target aggregation features corresponding to each pixel.

[0078] The depth map of the reference image in the multi-view image is generated based on the target aggregated features of each pixel.

[0079] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0080] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A method for generating a depth map based on multi-view stereo vision, characterized in that: The method comprises: Perform image segmentation on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment; Use a fast segmentation arbitrary model to segment a set of input multi-view images according to a preset RGB image source to generate a number of multi-level image segments and the confidence level corresponding to each image segment; The semantic features of the retained image segments and the contextual semantic features of the multi-view images are obtained by using a pre-trained visual language large model, and the contextual semantic features are assigned to each pixel in each image segment; the semantic features of the retained image segments corresponding to the top three in the order of the maximum confidence values ​​in each level of image segment set are extracted by using a pre-trained visual language large model, and the contextual semantic features of the multi-view images are generated, and the contextual semantic features are assigned to each pixel in each image segment; According to the semantic features of the retained image fragment to which each pixel belongs and the contextual semantic features, multi-level semantic feature aggregation is performed to obtain the semantic aggregation features of each pixel; according to the semantic features of the retained image fragment to which each pixel belongs and the contextual semantic features, multi-level semantic feature aggregation is performed through a fast segmentation arbitrary model to obtain the semantic aggregation features of each pixel: in, The semantic aggregation feature for each pixel, To quickly segment any pair of models, For fast segmentation, any model detects confidence in each retained image segment. is the global information of all image segments, For each preserved image segment, is element-wise multiplication, MLP[·] is a multi-layer perceptron; Aggregate the semantic aggregation features and appearance features of each pixel to obtain the target aggregation features corresponding to each pixel; The depth map of the reference image in the multi-view image is generated based on the target aggregated features of each pixel.

2. The method according to claim 1, characterized in that Before the step of aggregating the semantic aggregation features and the appearance features of each pixel to obtain the target aggregation features corresponding to each pixel, the method further includes: Construct a channel conversion network; the channel conversion network includes three cross-attention layers: in, is the cross attention vector, is the query vector, is the key vector, is a value vector, They are the learning weights of the query vector, the learning weights of the key vector, and the learning weights of the value vector.

3. The method according to claim 2, characterized in that The semantic aggregation features and appearance features of each pixel are aggregated to obtain the target aggregation features corresponding to each pixel, including: The semantic aggregation features of each pixel and the appearance features of the multi-view image are input into the channel conversion network for feature aggregation to obtain the target aggregation features corresponding to each pixel: in, is the target aggregation feature, is the l-th layer feature map on the c-th channel of the m-th subnetwork, is the average of all activations at all pixel locations in the current image segment, is the scaling factor of the channel conversion subnetwork, is the offset of the channel conversion subnetwork, is the standard deviation of all activations at all pixel locations in the current image segment, and ϵ is a constant.

4. The method according to claim 3, characterized in that Generate a depth map of the reference image in the multi-view image based on the target aggregation features of each pixel, including: A 3D cost volume is constructed according to the target aggregation feature of each pixel point, and after the channel conversion network is trained by a supervised learning method, an optimized target aggregation feature is output, and the optimized target aggregation feature generates a depth map of a reference image in a multi-view image through a 3D point cloud.

5. A depth map generation device based on multi-view stereo vision, characterized in that: The device comprises: An image segmentation module is used to perform image segmentation on a set of input multi-view images to obtain a number of image segments and the confidence level corresponding to each image segment; a fast segmentation arbitrary model is used to segment a set of input multi-view images according to a preset RGB image source to generate a number of multi-level image segments and the confidence level corresponding to each image segment; A semantic feature extraction module, used to obtain semantic features of the retained image segments and contextual semantic features of the multi-view images through a pre-trained visual language large model, and assign the contextual semantic features to each pixel in each image segment; extract semantic features of the top three retained image segments corresponding to the maximum confidence values ​​in each level of image segment set through a pre-trained visual language large model, and generate contextual semantic features of the multi-view images, and assign the contextual semantic features to each pixel in each image segment; The semantic feature aggregation module is used to perform multi-level semantic feature aggregation according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features to obtain the semantic aggregation features of each pixel; according to the semantic features of the retained image segment to which each pixel belongs and the contextual semantic features, perform multi-level semantic feature aggregation through a fast segmentation arbitrary model to obtain the semantic aggregation features of each pixel: in, The semantic aggregation feature for each pixel, To quickly segment any pair of models, For fast segmentation, any model detects confidence in each retained image segment. is the global information of all image segments, For each preserved image segment, is element-wise multiplication, MLP[·] is a multi-layer perceptron; The semantic-appearance feature aggregation module is used to aggregate the semantic aggregation features and appearance features of each pixel to obtain the target aggregation features corresponding to each pixel; The depth map aggregation module generates the depth map of the reference image in the multi-view image based on the target aggregation features of each pixel.

6. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

Citation Information

Patent Citations

  • Semantic perception multi-view terrain three-dimensional reconstruction method, medium and equipment

    CN118608708A

  • Image processing method and terminal device

    WO2023246822A1