A multi-view instance semantic alignment and labeling method and device, a terminal and a medium

Abstract

This invention discloses a method, apparatus, terminal, and medium for multi-view instance semantic alignment and annotation. The method acquires a multi-view image sequence of a scene to be processed, generates a single-instance positive sample mask set corresponding to each image in the multi-view image sequence; uses inference prompts to drive a large language model to perform label inference on the single-instance positive samples, determining an initial label set; uses inference verification prompts to drive the large language model to verify the initial label set, determining a global label set; uses selection prompts to drive the large language model to determine an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set; and performs error correction on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed. Therefore, it can effectively solve the problem of insufficient accuracy and inconsistent descriptions when processing videos and multi-view continuous images, where the same object may be labeled differently in different frames due to factors such as angle and lighting.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and more particularly to a method, apparatus, terminal, and medium for multi-view instance semantic alignment and annotation. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision, image description generation (image captioning) technology, as a bridge connecting the two, is widely used in scenarios such as video subtitles and intelligent monitoring. Its core is to automatically generate accurate and coherent natural language descriptions based on images or continuous images.

[0003] While progress has been made in the "image description" technology for single-frame static images, when processing videos and multi-view continuous images, the same object may be labeled with different tags in different frames due to factors such as angle and lighting. The descriptions of actions or scenes may also be consistent or contradictory, resulting in problems of insufficient accuracy and inconsistent descriptions.

[0004] Therefore, existing technologies still need improvement and development. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a multi-view instance semantic alignment and annotation method, device, terminal and medium to address the above-mentioned defects of the prior art. The aim is to solve the problem that when the prior art processes video and multi-view continuous images, the same object may be marked with different labels in different frames due to different angles, lighting and other factors, resulting in insufficient accuracy and inconsistent descriptions.

[0006] The technical solution adopted by this invention to solve the problem is as follows: In a first aspect, embodiments of the present invention provide a multi-view instance semantic alignment and annotation method, wherein the method includes: Obtain a multi-view image sequence of the scene to be processed, and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence; Obtain inference prompts, and use the inference prompts to drive a large language model to perform label inference on the single instance positive sample to determine the initial label set; Obtain inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set, thereby determining the global tag set; The selection prompt words are obtained, and the selection prompt words are used to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set. Error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed.

[0007] In one implementation method, generating a single-instance positive sample mask set corresponding to each image in the multi-view image sequence includes: Generate an initial mask set corresponding to each image in the multi-view image sequence; Redundant and nested masks are removed from the initial mask set to determine the single-instance positive sample mask set corresponding to each image.

[0008] In one implementation method, the inference prompt word is used to drive a large language model to perform label inference on the single instance positive sample, determining an initial label set, including: A preset sampling rate is obtained, and the multi-view image sequence is sampled according to the preset sampling rate to determine the multi-view image sub-sequence; Determine the single-instance positive sample mask corresponding to each image in the multi-view image sequence as a subset of the single-instance positive sample mask; The inference prompt words are used to drive a large language model to perform label inference on the single-instance positive sample mask subset to determine the initial label set.

[0009] In one implementation method, the inference prompt word is used to drive a large language model to perform label inference on the single-instance positive sample mask subset to determine an initial label set, including: Noise filtering and context anchoring are performed on the single-instance positive sample mask subset to determine the preprocessed mask set; Generate candidate labels for each mask in the preprocessed mask set; Determine whether the candidate label corresponding to each mask matches the label in the historical label set; If a match is found, a mapping relationship is established between the tags in the historical tag set and the mask. If there is no match, a mapping relationship is established between the candidate tag and the mask, and the candidate tag is appended to the historical tag set; Use the historical tag set as the initial tag set.

[0010] In one implementation method, the initial tag set is validated using the inference verification word-driven large language model to determine the global tag set, including: Check whether the mapping relationship between the mask in the single-instance positive sample mask subset and the labels in the initial label set is correct; If correct, the similarity between masks that have a mapping relationship with the same label in the initial label set is detected, and the label corresponding to the mask is determined based on the similarity. If incorrect, then determine the error mask subset, and use the inference prompt words to drive the large language model to perform label inference on the error mask subset, and update the initial label set; The global tag set is determined based on the tags corresponding to each mask.

[0011] In one implementation, the selected prompt word-driven large language model determines an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set, including: Check if there are any tags in the global tag set that match the masks in the single-instance positive sample mask set; If it exists, the matching tag will be used as the tag of the mask; If it does not exist, then find the label corresponding to the mask based on the candidate label set; The initial semantic segmentation map is determined based on each mask and corresponding label in the single-instance positive sample mask set.

[0012] In one implementation method, error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed, including: Global instances are determined by associating each mask in the initial semantic segmentation graph based on multi-view geometric constraints. The historical tags of each global instance are statistically analyzed, and the target tags corresponding to each global instance are selected. The target semantic segmentation map is determined by reversing the labels of each global instance based on the target label.

[0013] Secondly, embodiments of the present invention also provide a multi-view instance semantic alignment and annotation apparatus, wherein the multi-view instance semantic alignment and annotation apparatus includes: The mask generation module is used to acquire a multi-view image sequence of the scene to be processed and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence. The label inference module is used to obtain inference prompt words, and use the inference prompt words to drive the large language model to perform label inference on the single instance positive sample to determine the initial label set. The tag verification module is used to obtain inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set and determine the global tag set; The label selection module is used to obtain selection prompts and use the selection prompts to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set. The error correction module is used to correct errors in the initial semantic segmentation map and determine the target semantic segmentation map corresponding to the scene to be processed.

[0014] Thirdly, embodiments of the present invention also provide a terminal, the terminal including a memory and one or more processors; the memory stores one or more programs; the programs include instructions for executing the multi-view instance semantic alignment and annotation methods as described above; the processor is used to execute the programs.

[0015] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions, wherein the instructions are adapted to be loaded and executed by a processor to implement any of the multi-view instance semantic alignment and annotation methods described above.

[0016] The beneficial effects of this invention are as follows: In this embodiment, by acquiring a multi-view image sequence of the scene to be processed, a single-instance positive sample mask set corresponding to each image in the multi-view image sequence is generated; an inference prompt word-driven large language model is used to perform label inference on the single-instance positive samples to determine an initial label set; an inference verification word-driven large language model is used to verify the initial label set to determine a global label set; a selection prompt word-driven large language model is used to determine an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set; and error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed. Therefore, this invention effectively solves the problem in existing technologies for processing videos and multi-view continuous images where the same object may be labeled differently in different frames due to factors such as angle and lighting, resulting in insufficient accuracy and inconsistent descriptions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the multi-view instance semantic alignment and annotation method provided in this embodiment of the invention.

[0019] Figure 2 This is a flowchart illustrating a specific embodiment of the multi-view instance semantic alignment and annotation method provided in this invention.

[0020] Figure 3 This is a comparison diagram of multi-view instance consistency semantic alignment provided in the embodiments of the present invention.

[0021] Figure 4 This is a schematic diagram of the internal modules of the multi-view instance semantic alignment and annotation device provided in the embodiments of the present invention.

[0022] Figure 5 This is a schematic diagram of the terminal provided in the embodiment of the present invention. Detailed Implementation

[0023] This invention discloses a method, apparatus, terminal, and medium for multi-view instance semantic alignment and annotation. To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention.

[0024] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0025] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0026] With the rapid development of artificial intelligence and computer vision, image description generation (image captioning) technology, as a bridge connecting the two, is widely used in scenarios such as video subtitles and intelligent monitoring. Its core is to automatically generate accurate and coherent natural language descriptions based on images or continuous images.

[0027] While progress has been made in the "image description" technology for single-frame static images, when processing videos and multi-view continuous images, the same object may be labeled with different tags in different frames due to factors such as angle and lighting. The descriptions of actions or scenes may also be consistent or contradictory, resulting in problems of insufficient accuracy and inconsistent descriptions.

[0028] To address the aforementioned shortcomings of existing technologies, this invention provides a multi-view instance semantic alignment and annotation method. The method acquires a multi-view image sequence of the scene to be processed, generates a single-instance positive sample mask set corresponding to each image in the multi-view image sequence; uses inference prompts to drive a large language model to perform label inference on the single-instance positive samples, determining an initial label set; uses inference verification prompts to drive the large language model to verify the initial label set, determining a global label set; uses selection prompts to drive the large language model to determine an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set; and performs error correction on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed. Therefore, it effectively solves the problem of insufficient accuracy and inconsistent descriptions when processing videos and multi-view continuous images, where the same object may be labeled differently in different frames due to factors such as angle and lighting.

[0029] Exemplary method: like Figure 1 As shown, the method includes: Step 100: Obtain a multi-view image sequence of the scene to be processed, and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence.

[0030] The scene to be processed is any scene requiring semantic segmentation. The multi-view image sequence is a sequence of images acquired from different perspectives (including different heights, ranges, and angles) and at different shooting times for the scene to be processed. The multi-view image sequence can be a pre-processed image set or a set of images that have not undergone pre-processing or other processing. For example... Figure 2 As shown in step 1, in this embodiment, images of the scene to be processed are acquired at a constant rate of 30 frames per second from multiple angles and directions using a visual sensor. During acquisition, to ensure the validity of the input data and the efficiency of subsequent data processing, images are filtered using structural similarity (SSIM), and images with greater than 80% similarity between adjacent frames are discarded. The filtered images are represented as a multi-view image sequence, as shown below. , From the perspective of The total number of viewpoints.

[0031] Based on multi-view image sequences, a mask is generated for each image, with the target area being valid pixels and the background being invalid pixels. This accurately defines the spatial range of the target in the multi-view image. By masking irrelevant backgrounds and retaining valid areas, downstream tasks can focus on core information, improving processing efficiency and accuracy.

[0032] In one implementation, generating a single-instance positive sample mask set corresponding to each image in the multi-view image sequence includes: Step 101: Generate the initial mask set corresponding to each image in the multi-view image sequence; Step 102: Remove redundant and nested masks from the initial mask set to determine the single-instance positive sample mask set corresponding to each image.

[0033] For each image in a multi-view image sequence, an initial mask set corresponding to each image is first generated. Methods for generating image masks include traditional image processing methods and deep learning methods. For example, mask generation can be achieved through pixel grayscale, color, edge, and contour features, or based on a pre-trained model. Figure 2 As shown in step 2, this embodiment uses the SAM (Segment Anything Model) model to generate an initial mask for each frame of image, thereby obtaining the initial mask set corresponding to the multi-view image sequence.

[0034] For the initial mask set, redundant and nested masks are removed to obtain a single-instance positive sample mask set, thereby improving the efficiency of subsequent processing. This embodiment uses the Mask Generation and Filtering (GF) strategy to remove redundant and nested masks, obtaining a single-instance positive sample mask set, achieving redundancy-free filtering of the initial mask set, and ensuring that subsequent processing focuses on a single instance region. The specific implementation process includes: For each image in a multi-view image sequence Generate a single-instance positive sample mask set To address the issue of overlapping or nested generated masks, a mask containing other masks is defined as the negative sample set. The calculation formula is as follows: , in, This represents the number of pixels in the mask (i.e., the size of the mask). This represents the intersection (overlapping pixel area) of two masks. This is a pixel ratio threshold used to quantify the degree of significant inclusion. For Each of them If at least one other mask exists , making Include The pixel ratio threshold is greater than (Right now )and Then determine The redundant parent mask is classified as a negative sample.

[0035] A mask containing a single instance is considered a single-instance positive sample mask, represented as... ,satisfy and This allows for the creation of the initial mask set. Redundant filtering. Correspondingly, single-instance positive sample mask. It can be obtained through the complement operation, as shown below: .

[0036] Step 200: Obtain inference prompts and use the inference prompts to drive the large language model to perform label inference on the single instance positive sample to determine the initial label set.

[0037] Large language models, with Transformer as their core architecture, are capable of understanding the semantics and context of natural language, enabling natural language processing tasks such as text generation and logical reasoning. (Inference prompts) This tool is used to define the inference task type, clarify the inference steps, and provide output specifications for large language models. Inference prompts can be obtained through pre-setting or user input. After obtaining the inference prompts, the tool is used to drive the large language model to perform label inference on single-instance positive samples, resulting in an initial label set.

[0038] In one implementation, the inference prompt word drives a large language model to perform label inference on the single-instance positive sample, determining an initial label set, including: Step 201: Obtain a preset sampling rate, and sample the multi-view image sequence according to the preset sampling rate to determine the multi-view image sub-sequence; Step 202: Determine the single-instance positive sample mask corresponding to each image in the multi-view image sequence as a subset of the single-instance positive sample mask; Step 203: Use the inference prompt words to drive the large language model to perform label inference on the single instance positive sample mask subset to determine the initial label set.

[0039] To reduce computational costs and utilize temporal information, a sparse sampling strategy is employed based on the predicted sampling rate to select multi-view image subsequences from the multi-view image sequence. : , in, This is the preset sampling rate.

[0040] The single-instance positive sample mask corresponding to each image in the sampled multi-view image subsequence is used as a single-instance positive sample mask subset. An inference prompt word-driven large language model is used to perform label inference on the single-instance positive sample mask subset to determine the initial label set. This reduces the data processing volume of the large language model and improves the generation speed of the initial label set.

[0041] In one implementation, the inference prompt word drives a large language model to perform label inference on the single-instance positive sample mask subset to determine an initial label set, including: Step 2031: Perform noise filtering and context anchoring on the single-instance positive sample mask subset to determine the preprocessed mask set; Step 2032: Generate candidate labels corresponding to each mask in the preprocessed mask set; Step 2033: Determine whether the candidate label corresponding to each mask matches the label in the historical label set; Step 2034: If a match is found, establish a mapping relationship between the tags in the matched historical tag set and the mask; Step 2035: If there is no match, establish a mapping relationship between the candidate tag and the mask, and append the candidate tag to the historical tag set; Step 2036: Use the historical tag set as the initial tag set.

[0042] A large language model driven by inference prompts (LLMs) is used to perform label inference on a subset of masked positive samples from a single instance, resulting in an initial label set, such as... Figure 2 As shown in step 3, the logic of the large language model executing the label inference process with inference prompts is as follows: Content analysis phase: Noise filtering is performed on a subset of single-instance positive sample masks, ignoring pure black background pixels within the single-instance positive sample mask region and focusing only on valid texture areas; context anchoring is then performed on the noise-filtered single-instance positive sample masks to obtain a preprocessed mask set. Context anchoring assists in resolving low-resolution or blurred regions by analyzing the spatial adjacency relationship between the target region (single-instance positive sample mask) and the surrounding environment (e.g., "attached to a wall" or "located on the floor").

[0043] Decision-making stage: Generating candidate labels for each mask in the preprocessed mask set. In this embodiment, candidate labels are generated based on the visual features (color, texture, shape, etc.) of each mask, determining the candidate label set. Incremental matching detection is performed based on the candidate label set. The candidate labels corresponding to the masks are compared with the historical label set. The system compares the candidate label with labels in the existing label set (or the historical label set) and calculates the first similarity of visual attributes between the candidate label and the labels in the historical label set. If the similarity is higher than a first preset similarity threshold, the candidate label corresponding to the mask is confirmed to match the label in the historical label set, and a mapping relationship is established between the mask and the matched label in the historical label set (i.e., reusing the label and index in the historical label set). If the first similarity is lower than the first preset similarity threshold, the candidate label corresponding to the mask is confirmed to not match the label in the historical label set, a mapping relationship is established between the mask and the corresponding candidate label, and the candidate label is appended to the end of the historical label set. This matching process follows the principle of uniqueness to avoid generating duplicate semantic entries.

[0044] Traverse the candidate label set or the single-instance positive sample mask subset. Use the final historical label set as the initial label set corresponding to the single-instance positive sample mask subset.

[0045] Step 300: Obtain the inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set and determine the global tag set.

[0046] In short, reasoning verification words These are keywords or phrases, either preset or input by the user, used to verify and evaluate the validity of the reasoning process and results of the large language model. This embodiment utilizes verification prompts to drive the large language model to process the initial tag set (or the updated historical tag set). Closed-loop verification is performed to ultimately generate a deterministic global tag set that is deduplicated and covers the entire scenario. .

[0047] In one implementation, the inference verification word-driven large language model is used to verify the initial tag set to determine the global tag set, including: Step 301: Check whether the mapping relationship between the mask in the single-instance positive sample mask subset and the labels in the initial label set is correct; Step 302: If correct, detect the similarity between masks in the initial label set that have a mapping relationship with the same label, and determine the label corresponding to the mask based on the similarity. Step 303: If incorrect, determine the error mask subset, and use the inference prompt words to drive the large language model to perform label inference on the error mask subset, and update the initial label set; Step 304: Determine the global tag set based on the tags corresponding to each mask.

[0048] An inference-driven large language model is used to validate the initial tag set and determine the global tag set, such as... Figure 2 As shown in step 4, the logic for the inference verification word-driven large language model to perform label verification is as follows: Two-way verification: Traverse the tags in the initial tag set and check whether the mapping relationship between the mask in the single-instance positive sample mask subset and the tags in the initial tag set is correct. Specifically, this includes: checking whether each single-instance positive sample mask has a mapping relationship with a tag (i.e., whether it has been assigned a tag), and checking whether each tag in the initial tag set has a mapping relationship with at least one single-instance positive sample mask. If both of the above conditions are met, the mapping relationship between the mask in the single-instance positive sample mask subset and the tags in the initial tag set is confirmed to be correct; if one of the conditions is not met, the mapping relationship is confirmed to be incorrect.

[0049] Visual disambiguation: If the above verification result is correct, the similarity between multiple masks that have a mapping relationship with the same label in the initial label set is detected as the second similarity. The second similarity is used to measure the similarity of visual attributes (color, shape) between each mask. If the second similarity is higher than or equal to the second preset similarity threshold, they are identified as different instances of the same category, and the label is reused; if the second similarity is lower than the second similarity threshold, it indicates that there is a significant difference in visual features between the masks, and fine-grained labels (such as "wooden chair" and "sofa chair") are generated for differentiation.

[0050] Residual Inference: If the above verification result is incorrect, the masks with incorrect mapping relationships are extracted to generate an error mask subset. Inference prompts are then used to drive the large language model to perform label inference on this error mask subset, updating the initial label set. Subsequently, inference verification prompts are used again to drive the large language model to verify the updated initial label set until no masks with incorrect mapping relationships remain.

[0051] The global label set is determined based on the labels corresponding to each mask obtained at the end (i.e., there are no masks with incorrect mapping relationships). This embodiment ensures that each label in the initial label set is mapped to a valid visual segment through two-way verification, visual disambiguation, and residual inference mechanisms, and supplements any missing semantic entities, ultimately forming a determined global label set.

[0052] Step 400: Obtain selection prompts and use the selection prompts to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set.

[0053] Select prompt word These are preset or user-inputted tags used to accurately match corresponding labels to each mask. This embodiment uses a selection prompt word-driven large language model to extend the processing scope to all viewpoint images, consistently mapping labels from the global label set to masks from all viewpoints.

[0054] In one implementation, the selected prompt word-driven large language model determines an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set, including: Step 401: Check if there are any tags in the global tag set that match the masks in the single-instance positive sample mask set; Step 402: If it exists, use the matching tag as the tag of the mask; Step 403: If it does not exist, then search for the label corresponding to the mask based on the candidate label set; Step 404: Determine the initial semantic segmentation map based on each mask and corresponding label in the single instance positive sample mask set.

[0055] A selection prompt word-driven large language model is used to select mask labels based on a global label set. For example... Figure 2 As shown in step 5, the execution logic of the prompt word-driven large language model is as follows: The model iterates through all masks in the single-instance positive sample mask set and scans the global label set for visual re-identification. Specifically, it performs a fine-grained comparison between the visual features of the single-instance positive sample mask and the masks corresponding to the labels in the global label set. If a match with consistent visual features is found, the label or global index of the matching mask is reused without generating new text. If no mask visually matches the single-instance positive sample mask in the global label set, the large language model queries the candidate label set. When a corresponding label is found in the candidate label set, a state transition operation is performed: the label and its visual features are registered in the global label set, and a new global index is assigned. This operation avoids matching based solely on text labels (e.g., avoiding grouping doors of different colors into the same index simply because the label is "door"), and enforces "visual-semantic coupling," thereby ensuring consistency across perspectives.

[0056] Step 500: Perform error correction on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed.

[0057] In simple terms, global mask-label error correction is performed based on the initial semantic segmentation map, and local misclassifications are corrected using multi-view observation data to generate the final instance-consistent target semantic segmentation map.

[0058] In one implementation, error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed, including: Step 501: Based on multi-view geometric constraints, associate each mask in the initial semantic segmentation graph to determine the global instance; Step 502: Statistically analyze the historical tags of each global instance and select the target tags corresponding to each global instance; Step 503: Correct the labels of each global instance in reverse according to the target label to determine the target semantic segmentation map.

[0059] like Figure 2 As shown in step 6, based on multi-view geometric constraints (bounding box intersection-over-union ratio and centroid distance), the masks corresponding to spatial locations under different views in the initial semantic segmentation map are associated as global instances. For each global instance Statistical analysis was performed on historical tags, and the optimal tag was selected using a voting mechanism. This is used as the target label, and the initial semantic segmentation map under all relevant perspectives is then corrected using this target label. This eliminates classification noise from local viewpoints. The specific calculation method is as follows: For each global instance Collect its data from all observation perspectives The assigned tags (or tag indices) form a tag set (or tag index set). Calculate the mode of the tag set, and use it as the final target tag for this instance. Then, iterate through all perspectives involved in this global instance. Update the semantic segmentation map based on the target label. This step utilizes redundant observation information from multiple perspectives to automatically correct occasional misclassifications in a single frame caused by occlusion, changes in lighting, or model randomness.

[0060] Figure 3 The image shows a comparison of the semantic consistency of our method and the CLIP (contrast-language pre-training) method within the closed set of labels inferred from Replica (a dataset). It can be seen that our method maintains consistent semantics for the same instance even when the scene perspective changes, while the CLIP model falls far short of this performance.

[0061] Based on the above embodiments, the present invention also provides a multi-view instance semantic alignment and annotation device, such as... Figure 4 As shown, the device includes: Mask generation module 01 is used to acquire a multi-view image sequence of the scene to be processed and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence. The label inference module 02 is used to obtain inference prompt words, and use the inference prompt words to drive the large language model to perform label inference on the single instance positive sample to determine the initial label set; Tag verification module 03 is used to obtain inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set and determine the global tag set; The label selection module 04 is used to obtain selection prompt words and use the selection prompt words to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set. Error correction module 05 is used to perform error correction on the initial semantic segmentation map and determine the target semantic segmentation map corresponding to the scene to be processed.

[0062] Based on the above embodiments, the present invention also provides a terminal, the principle block diagram of which can be as follows: Figure 5As shown, the terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides the environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-view instance semantic alignment and annotation method. The display screen can be an LCD screen or an e-ink screen.

[0063] Those skilled in the art will understand that Figure 5 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In one implementation, the terminal's memory stores one or more programs, and these programs are configured to be executed by one or more processors, and the programs contain instructions for performing multi-view instance semantic alignment and annotation methods.

[0065] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0066] In summary, this invention discloses a method, apparatus, terminal, and medium for multi-view instance semantic alignment and annotation. The method acquires a multi-view image sequence of the scene to be processed, generates a single-instance positive sample mask set corresponding to each image in the multi-view image sequence; uses inference prompts to drive a large language model to perform label inference on the single-instance positive samples, determining an initial label set; uses inference verification prompts to drive the large language model to verify the initial label set, determining a global label set; uses selection prompts to drive the large language model to determine an initial semantic segmentation map based on the single-instance positive sample mask set and the global label set; and performs error correction on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed. Therefore, it can effectively solve the problem of insufficient accuracy and inconsistent descriptions in existing technologies when processing video and multi-view continuous images, where the same object may be labeled differently in different frames due to factors such as angle and lighting.

[0067] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A multi-view instance semantic alignment and annotation method, characterized in that, The method includes: Obtain a multi-view image sequence of the scene to be processed, and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence; Obtain inference prompts, and use the inference prompts to drive a large language model to perform label inference on the single instance positive sample to determine the initial label set; Obtain inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set, thereby determining the global tag set; The selection prompt words are obtained, and the selection prompt words are used to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set. Error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed.

2. The multi-view instance semantic alignment and annotation method according to claim 1, characterized in that, Generating a single-instance positive sample mask set corresponding to each image in the multi-view image sequence includes: Generate an initial mask set corresponding to each image in the multi-view image sequence; Redundant and nested masks are removed from the initial mask set to determine the single-instance positive sample mask set corresponding to each image.

3. The multi-view instance semantic alignment and annotation method according to claim 1, characterized in that, The inference prompt words are used to drive a large language model to perform label inference on the single instance positive sample, and an initial label set is determined, including: A preset sampling rate is obtained, and the multi-view image sequence is sampled according to the preset sampling rate to determine the multi-view image sub-sequence; Determine the single-instance positive sample mask corresponding to each image in the multi-view image sequence as a subset of the single-instance positive sample mask; The inference prompt words are used to drive a large language model to perform label inference on the single-instance positive sample mask subset to determine the initial label set.

4. The multi-view instance semantic alignment and annotation method according to claim 3, characterized in that, The inference prompt word-driven large language model is used to perform label inference on the single-instance positive sample mask subset to determine the initial label set, including: Noise filtering and context anchoring are performed on the single-instance positive sample mask subset to determine the preprocessed mask set; Generate candidate labels for each mask in the preprocessed mask set; Determine whether the candidate label corresponding to each mask matches the label in the historical label set; If a match is found, a mapping relationship is established between the tags in the historical tag set and the mask. If there is no match, a mapping relationship is established between the candidate tag and the mask, and the candidate tag is appended to the historical tag set; Use the historical tag set as the initial tag set.

5. The multi-view instance semantic alignment and annotation method according to claim 3, characterized in that, The initial tag set is validated using the inference verification word-driven large language model to determine the global tag set, including: Check whether the mapping relationship between the mask in the single-instance positive sample mask subset and the labels in the initial label set is correct; If correct, the similarity between masks that have a mapping relationship with the same label in the initial label set is detected, and the label corresponding to the mask is determined based on the similarity. If incorrect, then determine the error mask subset, and use the inference prompt words to drive the large language model to perform label inference on the error mask subset, and update the initial label set; The global tag set is determined based on the tags corresponding to each mask.

6. The multi-view instance semantic alignment and annotation method according to claim 1, characterized in that, The selected prompt word-driven large language model determines the initial semantic segmentation map based on the single-instance positive sample mask set and the global label set, including: Check if there are any tags in the global tag set that match the masks in the single-instance positive sample mask set; If it exists, the matching tag will be used as the tag of the mask; If it does not exist, then find the label corresponding to the mask based on the candidate label set; The initial semantic segmentation map is determined based on each mask and corresponding label in the single-instance positive sample mask set.

7. The multi-view instance semantic alignment and annotation method according to claim 1, characterized in that, Error correction is performed on the initial semantic segmentation map to determine the target semantic segmentation map corresponding to the scene to be processed, including: Global instances are determined by associating each mask in the initial semantic segmentation graph based on multi-view geometric constraints. The historical tags of each global instance are statistically analyzed, and the target tags corresponding to each global instance are selected. The target semantic segmentation map is determined by reversing the labels of each global instance based on the target label.

8. A multi-view instance semantic alignment and annotation device, characterized in that, The device includes: The mask generation module is used to acquire a multi-view image sequence of the scene to be processed and generate a single-instance positive sample mask set corresponding to each image in the multi-view image sequence. The label inference module is used to obtain inference prompt words, and use the inference prompt words to drive the large language model to perform label inference on the single instance positive sample to determine the initial label set. The tag verification module is used to obtain inference verification words, and use the inference verification words to drive the large language model to verify the initial tag set and determine the global tag set; The label selection module is used to obtain selection prompts and use the selection prompts to drive the large language model to determine the initial semantic segmentation map based on the single instance positive sample mask set and the global label set. The error correction module is used to correct errors in the initial semantic segmentation map and determine the target semantic segmentation map corresponding to the scene to be processed.

9. A terminal, characterized in that, The terminal includes a memory and one or more processors; the memory stores one or more programs; the programs contain instructions for executing the multi-view instance semantic alignment and annotation method as described in any one of claims 1-7; the processor is used to execute the programs.

10. A computer-readable storage medium storing a plurality of instructions thereon, characterized in that, The instructions are loaded and executed by the processor to implement the steps of the multi-view instance semantic alignment and annotation method as described in any one of claims 1-7.

Images