An open-vocabulary panoramic image segmentation method, device and electronic equipment based on visual-linguistic cue collaboration

Abstract

The application provides an open vocabulary panoramic image segmentation method and device based on visual-language prompt cooperation and an electronic device. The method comprises the following steps: acquiring a panoramic image to be segmented and a target category name set and inputting an open vocabulary mask segmentation model to generate a category segmentation result of the panoramic image; the model performs the following steps: S1, performing feature extraction on the panoramic image to generate a candidate mask and corresponding mask query features; S2, performing text encoding on each target category name to generate target category text features; S3, querying a plurality of candidate region visual prompts matched with the target category text features from a source domain visual prompt library; S4, calculating a compatible gating value and a cooperative visual feature; S5, calculating a text classification item of each candidate mask and a visual residual item modulated by the compatible gating value to obtain a text-visual cooperative classification score; and S6, performing category allocation according to each candidate mask and the corresponding text-visual cooperative classification score to generate a category segmentation result.

Description

Technical Field

[0001] This invention relates to the technical fields of panoramic image processing, computer vision, visual language processing, and open-vocabulary semantic segmentation, specifically to an open-vocabulary panoramic image segmentation method, apparatus, and electronic device based on visual-language cues collaboration. Background Technology

[0002] Isometric cylindrical projection panoramic images can cover a 360-degree horizontal scene in a single image, and can be used for autonomous driving environment understanding, robot navigation, indoor spatial perception, digital twins, and immersive scene understanding. Compared with ordinary perspective images, isometric cylindrical projection images have non-uniform deformation that varies with latitude, with significant stretching in high-latitude regions, periodic continuity at the left and right boundaries, and visual discontinuity for targets crossing the boundaries. Therefore, segmentation models trained directly using ordinary field-of-view images are prone to geometric domain differences.

[0003] Open-ended panoramic image segmentation also requires the model to perform pixel-level classification of panoramic images based on the target category names given during the deployment phase, even when the target category set is not entirely consistent with the source domain training category set. Target categories may include categories not present during training, or they may have aliases, hierarchical relationships, semantic correlations, or differences in category granularity with the source domain categories. Since pixel-by-pixel annotation of the target panoramic image is costly, training using only annotated ordinary field-of-view source domain images and directly deploying them to the target panoramic image is a technical approach with practical application value.

[0004] Existing open panoramic segmentation techniques primarily focus on the geometric transfer between ordinary field-of-view images and equidistant cylindrical projection panoramic images, while mask classification typically still relies mainly on textual representations formed by category names. When relying solely on textual representations, the local appearance, context, scale, and boundary information already contained in the pixel-level labeled regions of the source domain are not fully utilized. On the other hand, if the visual features of the ordinary field-of-view source domain region are directly used to replace the target category text prototype, a geometrical distribution mismatch between visual evidence and panoramic mask queries will occur due to the lack of latitude-varying panoramic distortion in ordinary field-of-view visual cues. Furthermore, when the source and target domain vocabularies are inconsistent, the source category bias is easily amplified.

[0005] Therefore, there is an urgent need in this field for an open panoramic image segmentation technology that can match local visual evidence in the source domain with the panoramic geometric distribution without using the target panoramic image to construct or update the visual cue library, and allows the target category text to control the retrieval range and intensity of the source domain visual evidence, while preserving the main path of open vocabulary recognition of the text. Summary of the Invention

[0006] To overcome the problems existing in related technologies, this invention proposes an open vocabulary panoramic image segmentation method, device, and electronic device based on visual-language cues collaboration.

[0007] Specifically, the present invention is achieved through the following technical solution:

[0008] According to a first aspect of the embodiments of this specification, an open vocabulary panoramic image segmentation method based on visual-language cue collaboration is provided, the method comprising:

[0009] Obtain the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names, and input them into a trained open-vocabulary mask segmentation model. The model then generates the category segmentation result for the panoramic image. The open-vocabulary mask segmentation model performs the following steps:

[0010] Step S1: Extract features from the panoramic image to generate at least one candidate mask and its corresponding mask query feature;

[0011] Step S2: Text encoding is performed on each of the target category names to generate target category text features;

[0012] Step S3: Query multiple candidate region visual cues that match the text features of the target category from the source domain visual cue library; the source domain visual cue library is built offline and kept frozen during the training phase, and it stores one or more visual cue slots corresponding to each source domain category, and each visual cue slot contains at least the region visual features corresponding to the source domain perspective sample image.

[0013] Step S4: Generate a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features; and fuse the multiple candidate region visual cues based on the target category text features to generate collaborative visual features.

[0014] Step S5: For each candidate mask, based on the similarity between its mask query features and the target category text features and the co-visual features, determine the text classification term and the visual residual term modulated by the compatibility gate value, and obtain the text visual co-classification score of the candidate mask relative to each target category based on the text classification term and the visual residual term.

[0015] Step S6: Assign categories based on each candidate mask and its corresponding text visual co-classification score to generate the category segmentation result.

[0016] According to a second aspect of the embodiments of this specification, an open-vocabulary panoramic image segmentation apparatus based on visual-language cueing collaboration is provided, the apparatus being configured to perform the method described in the first aspect, the apparatus comprising:

[0017] The data acquisition unit is used to acquire the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names, and input them into the trained open vocabulary mask segmentation model.

[0018] A data processing unit is configured to generate category segmentation results for the panoramic image using the model; wherein the open-vocabulary mask segmentation model performs the following steps:

[0019] Step S1: Extract features from the panoramic image to generate at least one candidate mask and its corresponding mask query feature;

[0020] Step S2: Text encoding is performed on each of the target category names to generate target category text features;

[0021] Step S3: Query multiple candidate region visual cues that match the text features of the target category from the source domain visual cue library; the source domain visual cue library is built offline and kept frozen during the training phase, and it stores one or more visual cue slots corresponding to each source domain category, and each visual cue slot contains at least the region visual features corresponding to the source domain perspective sample image.

[0022] Step S4: Generate a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features; and fuse the multiple candidate region visual cues based on the target category text features to generate collaborative visual features.

[0023] Step S5: For each candidate mask, based on the similarity between its mask query features and the target category text features and the co-visual features, determine the text classification term and the visual residual term modulated by the compatibility gate value, and obtain the text visual co-classification score of the candidate mask relative to each target category based on the text classification term and the visual residual term.

[0024] Step S6: Assign categories based on each candidate mask and its corresponding text visual co-classification score to generate the category segmentation result.

[0025] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in the first aspect.

[0026] The embodiments of the present invention have at least the following technical effects:

[0027] (1) This invention enables panoramic image segmentation with open vocabulary. The open vocabulary masking segmentation model of this invention generates target category text features by text encoding the target category name, and retrieves candidate region visual cues from the source domain visual cue library based on these text features. This allows the model to perform pixel-level classification based on any target category name given during the deployment phase, without requiring the target category name to be completely identical to the source domain category name. Thus, open category segmentation of panoramic images can still be achieved even when the target category set is not completely identical to the source domain training category set.

[0028] (2) The source domain visual cue library of the present invention is constructed offline and kept frozen during the training phase. It stores visual cue slots corresponding to each source domain category, and each visual cue slot contains the regional visual features corresponding to the source domain perspective sample image. Therefore, during the inference phase, source domain visual evidence can be obtained simply by retrieving it from the cue library. There is no need to construct or update the visual cue library using the panoramic image to be segmented, nor is it necessary to annotate or generate pseudo-labels for the target panoramic image.

[0029] (3) This invention generates a compatibility gating value based on the similarity between the overall visual features of the candidate region visual cues and the text features of the target category, and modulates the visual residual term using this compatibility gating value. When the compatibility between the source domain visual evidence and the target category text is low, the visual residual term is attenuated, and the influence of the source domain visual information on the classification result is limited, thereby reducing the risk of amplifying the source category bias when the source domain and target domain vocabularys are inconsistent. Therefore, this invention limits the strength of the source domain visual evidence through a compatibility gating mechanism, avoiding the amplification of the source category bias.

[0030] (4) This invention determines the text classification term based on the similarity between the mask query feature and the target category text feature, and determines the visual residual term based on the similarity between the mask query feature and the co-visual feature. The final classification score is obtained by combining the text classification term and the visual residual term. Thus, the text classification path always participates in the classification decision, and visual evidence is only added as an auxiliary correction term, avoiding the decline in open vocabulary recognition ability caused by completely replacing the text prototype with the source domain visual prototype.

[0031] (5) This invention fuses visual cues from multiple candidate regions based on the text features of the target category to generate collaborative visual features. Thus, the visual evidence retrieved from the source domain visual cue library is not used directly in a fixed manner, but is adaptively aggregated based on the text of the target category, so that the collaborative visual features can be dynamically adjusted according to the semantic requirements of the current target category.

[0032] In summary, the source domain visual cue library of this invention stores the regional visual features corresponding to source domain perspective sample images, while the inference stage processes isometric cylindrical projection panoramic images. Therefore, without using target panoramic images for training, open segmentation transfer from source domain perspective sample images to panoramic target images is achieved through text feature retrieval and gated modulation of collaborative visual features. Attached Figure Description

[0033] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0034] Figure 1 This is a schematic flowchart illustrating an open vocabulary panoramic image segmentation method based on visual-language cues collaboration, as shown in an exemplary embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the training and inference process of an open vocabulary masking segmentation model according to an exemplary embodiment of the present invention;

[0036] Figure 3 This is a schematic diagram illustrating the process of constructing a source domain visual cue library according to an exemplary embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram illustrating an example of multi-dimensional panoramic distortion synthesis according to an exemplary embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram illustrating a collaborative visual feature generation process according to an exemplary embodiment of the present invention;

[0039] Figure 6 This is a schematic diagram illustrating the training process of an open vocabulary masking segmentation model according to an exemplary embodiment of the present invention;

[0040] Figure 7 This is a schematic diagram illustrating the reasoning process of an open vocabulary masking segmentation model according to an exemplary embodiment of the present invention;

[0041] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present invention;

[0042] Figure 9 This is a block diagram illustrating an open vocabulary panoramic image segmentation device based on visual-language cues collaboration, as shown in an exemplary embodiment of the present invention. Detailed Implementation

[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention as detailed in the appended claims.

[0044] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular forms “a,” “the,” and “the” used in this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0045] To address the problems in existing open-vocabulary panoramic image segmentation, such as insufficient utilization of local visual evidence in the source domain, geometrical distribution mismatch between conventional perspective field visual cues and panoramic mask queries, the tendency of visual cues to introduce source category bias when the source and target domain vocabularies are inconsistent, and instability in prediction caused by projection changes, the technical solution of this invention enables the segmentation training view and the source domain visual cue library view to share a matching multi-dimensional panoramic distortion transform family, thereby simultaneously handling the geometrical distribution differences between visual cues and mask queries; it retrieves local visual evidence in the source domain using target category text features across the vocabulary, rather than requiring the target category to have the same name as the source domain category; and it always preserves the main path of text classification during processing, introducing source domain visual information only with bounded visual residuals that are compatible with gating, structurally limiting source category bias; in addition, it constrains the consistency between the source domain view and the distortion view through three levels: structure, query-cue relationship, and semantic output.

[0046] This invention provides an open vocabulary panoramic image segmentation method based on visual-language cues collaboration. Figure 1 This is a flowchart illustrating an open-vocabulary panoramic image segmentation method based on visual-language cues collaboration, as shown in an exemplary embodiment of the present invention. Figure 1 As shown, the open vocabulary panoramic image segmentation method includes:

[0047] Obtain the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names, and input them into the trained open vocabulary mask segmentation model.

[0048] The model generates the category segmentation results for the panoramic image.

[0049] The open vocabulary mask segmentation model performs the following steps:

[0050] Step S1: Extract features from the panoramic image to generate at least one candidate mask and its corresponding mask query feature;

[0051] Step S2: Text encoding is performed on each of the target category names to generate target category text features;

[0052] Step S3: Query multiple candidate region visual cues that match the text features of the target category from the source domain visual cue library; the source domain visual cue library is built offline and kept frozen during the training phase, and it stores one or more visual cue slots corresponding to each source domain category, and each visual cue slot contains at least the region visual features corresponding to the source domain perspective sample image.

[0053] Step S4: Generate a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features; and fuse the multiple candidate region visual cues based on the target category text features to generate collaborative visual features.

[0054] Step S5: For each candidate mask, based on the similarity between its mask query features and the target category text features and the co-visual features, determine the text classification term and the visual residual term modulated by the compatibility gate value, and obtain the text visual co-classification score of the candidate mask relative to each target category based on the text classification term and the visual residual term.

[0055] Step S6: Assign categories based on each candidate mask and its corresponding text visual co-classification score to generate the category segmentation result.

[0056] The open-vocabulary masking segmentation model of this invention performs text encoding on the target category name and retrieves candidate region visual cues from a source domain visual cue library built offline and kept frozen during the training phase, without requiring the target panoramic image to participate in training or update the cue library. A compatibility gate value is generated based on the similarity between the overall visual features of the candidate region visual cues and the text features of the target category. Co-visual features are generated by fusing candidate visual cues with text features as a condition. While retaining the text classification terms of the mask query features and the target category text features, the similarity between the mask query features and the co-visual features is modulated into a visual residual term using the compatibility gate value. The two are added together to obtain a text-visual co-classification score, ensuring that source domain visual evidence only participates in classification as a bounded auxiliary, automatically decaying when the compatibility is low, reducing the risk of source category bias. Pixel-level category segmentation results are also generated based on the candidate mask and the co-classification score. This model can achieve open-category segmentation of panoramic images without using the target panoramic image for training or cue library updates, and can also take into account the open-vocabulary recognition capability of the main text path.

[0057] Next, combined Figures 2 to 7 This invention provides a detailed description of the open-vocabulary panoramic image segmentation process in embodiments of the present invention.

[0058] To facilitate the description of the technical solution in this embodiment, multi-dimensional panoramic distortion synthesis is performed through a multi-latitude panoramic distortion synthesis component (ML-PDS); visual cue retrieval, aggregation, gating, and collaborative classification processes driven by target category text are performed through a text-visual prompt collaboration component (TVPC); and a training process for structural consistency, cue consistency, and semantic consistency between the source domain perspective sample image and the panoramic distortion view is performed through a distortion-consistent prompt adaptation component (DCPA). The above abbreviations are only used to facilitate the description of the technical functions of each part and do not constitute a limitation on the technical solution of this invention.

[0059] Among them, the multi-dimensional panoramic distortion synthesis component is used to generate source domain perspective sample images and panoramic distortion views with corresponding content but different projection forms, and form matching view pairs to provide panoramic source domain data for training open vocabulary mask segmentation models and building source domain visual cue libraries.

[0060] The text-visual cue collaboration component is used to retrieve and aggregate source domain visual cues based on target category text features, and participates in mask classification as visual residual terms modulated by compatible gating values.

[0061] The distortion consistency cue adaptation component uses the matching view pairs to apply consistency constraints to the mask structure, cue relationships, and semantic output of the open vocabulary mask segmentation model.

[0062] It is worth noting that the multi-dimensional panoramic distortion synthesis component and the distortion consistency cue adaptation component in the model are mainly executed during the training phase, while the text-visual cue collaboration component participates in model optimization during the training phase and is used to generate text-visual collaborative classification scores during the model inference phase.

[0063] like Figure 2As shown, this open-vocabulary masking segmentation model uses source domain perspective sample images with pixel-level category annotations as supervision during the training phase. A multi-dimensional panoramic distortion synthesis component generates a panoramic distortion view corresponding to the content of the source domain perspective sample images. A source domain visual cue library is constructed using the pixel-level category-annotated regions of the source domain perspective sample images. This source domain visual cue library is built offline and frozen during the training phase. During the model inference phase, the isometric cylindrical projection panoramic image to be segmented and the target category name set are used as input. Visual evidence is retrieved and aggregated from the source domain visual cue library using target category text features. While maintaining the text classification terms, a visual residual term modulated by a compatible gate value is formed to output the pixel-level category segmentation result.

[0064] In this embodiment, during model training, the labeled source domain dataset and source domain category set are first obtained. The labeled source domain dataset contains N sample pairs, and the i-th sample pair is obtained from the source domain perspective sample image. and corresponding pixel-level category labels The set of source domain categories is denoted as .

[0065] During the model inference phase, the system's target deployment task provides the isometric cylindrical projection panoramic image to be segmented. and the set of target category names Target category name set Can be combined with source domain category set different.

[0066] After obtaining the source domain dataset and the source domain category set, a source domain visual cue library can be constructed:

[0067] Based on each source domain category in the source domain category set, for the object-labeled region in the source domain perspective sample image containing objects of that source domain category, a first sub-image and a second sub-image are cropped respectively. The first sub-image includes the object region, and the second sub-image includes the object's neighborhood region. A panoramic distortion transformation is performed on the first sub-image and the second sub-image respectively to obtain a panoramic distortion view corresponding to each sub-image. The image encoder in the open vocabulary masking segmentation model is used to extract the regional visual features of the panoramic distortion view of each sub-image. According to the source domain category set, the regional visual features and their corresponding metadata are written into the corresponding visual cue slots of the source domain visual cue library.

[0068] In some embodiments, panoramic distortion transformation includes:

[0069] Obtain transformation parameters, including camera latitude, horizontal field of view, and effective area clipping parameters; on an equidistant cylindrical projection canvas, construct an inverse mapping relationship from pixels of the panoramic distortion view to pixels of the first sub-image or the second sub-image based on the transformation parameters; according to the inverse mapping relationship, resample and interpolate the first sub-image and the second sub-image respectively to obtain the corresponding panoramic distortion view.

[0070] The aforementioned panoramic distortion transformation (also known as multi-dimensional panoramic distortion synthesis) can be specifically implemented as follows: given a source domain perspective sample image. and its pixel-level category labeling sampling camera latitude Horizontal field of view and effective area clipping parameters The transformation parameters are then used to synchronously transform the source domain perspective sample image and its pixel-level category annotations to obtain a panoramic distortion view. and its synchronous distortion annotation :

[0071] (1)

[0072] in, and These are the source domain perspective sample images before transformation and their pixel-level category annotations; and These are panoramic distortion views obtained after panoramic distortion transformation and their synchronous distortion annotations, respectively. Transformation parameters Controlled synchronous panoramic distortion transformation; For the set of transformation parameters, To sample the camera latitude, For horizontal field of view, Parameters for effective region cropping. The image and annotations share the same set. The only difference is the resampling interpolation kernel.

[0073] In subsequent training, the source domain perspective sample image before transformation will be used. As the source domain view, the transformed image As a panoramic distortion view corresponding to its content; accordingly, and These serve as supervisory annotations for the two views, respectively. This results in paired samples of source domain views and panoramic distortion views with corresponding content but different projection forms.

[0074] In one specific implementation, the short side of the source domain perspective sample image can be scaled proportionally to 640 pixels; the horizontal field of view is sampled from 80 degrees, 90 degrees, and 100 degrees, the camera latitude is sampled from -30 degrees, -20 degrees, -10 degrees, 0 degrees, 10 degrees, 20 degrees, and 30 degrees, and the yaw angle is fixed at 0 degrees; an inverse ray mapping from the target pixel to the source domain perspective sample image is established on a 1280×2560 pixel equidistant cylindrical projection canvas. RGB images use bilinear interpolation, discrete category labels use nearest neighbor interpolation, invalid image positions are set to zero, and invalid label positions are assigned a value of 255.

[0075] After projection, find the effective pixel bounding box, expand it at a random ratio of 1.05 to 1.15, perform square filling, and scale to... Pixels. When the effective pixel ratio is below 0.45, resampling is performed up to 12 times, and if the condition is still not met, the candidate with the highest quality effective region is retained. To ensure reproducibility, a random seed and metadata such as latitude, field of view, clipping boundary, and effective ratio are saved for each source domain sample.

[0076] It should be noted that the effective area clipping parameters in this step... Used to crop, expand, and square the effective area after projection, and does not belong to the object focus clipping or context clipping type corresponding to the subsequent first and second sub-images.

[0077] To further illustrate the coordinate relationships in the reverse projection, let the height of the equidistant cylindrical projection canvas be . Width is The output pixel index is Then the longitude corresponding to that pixel and latitude for:

[0078] (2)

[0079] in, and These are the height and width of the equidistant cylindrical projection canvas, respectively. For the first Longitude corresponding to column pixels; For the first The dimension corresponding to the row pixels; Pi is the mathematical constant of a circle.

[0080] This mapping allows horizontal coverage from to The longitude range, and make the vertical direction cover from to The latitudinal range.

[0081] The output pixels are obtained based on the longitude and latitude. The corresponding spherical unit ray:

[0082] (3)

[0083] in, Indicates output pixels The corresponding spherical unit ray; the three components within the square brackets are the three coordinate components of this unit ray in the three-dimensional coordinate system.

[0084] The spherical unit ray is arranged according to the latitude of the sampling camera. Rotate the image to obtain the rotated 3D ray. Let the height of the scaled source domain perspective sample image be... Width is Then, with the horizontal field of view angle Corresponding vertical field of view for:

[0085] (4)

[0086] in, This refers to the vertical field of view. and These represent the height and width of the scaled source domain perspective sample image, respectively.

[0087] In this embodiment, the horizontal field of view is converted into the corresponding vertical field of view based on the aspect ratio of the source domain perspective sample image.

[0088] Let the rotated three-dimensional ray components be... , and A ray is considered valid only if it is directed towards the camera and is simultaneously within both the horizontal and vertical fields of view.

[0089] (5)

[0090] in, The first inequality is used to define the direction of the ray towards the front of the camera; the second inequality is used to define the ray within the horizontal field of view and the vertical field of view, respectively.

[0091] For an effective ray that satisfies formula (5), its continuous sampling coordinates on the source domain perspective sample image are... and for:

[0092] (6)

[0093] in, and These are the horizontal and vertical continuous sampling coordinates of the effective ray on the source domain perspective sample image, respectively; , and These are the rotated three-dimensional ray components; and These represent the width and height of the scaled source domain perspective sample image, respectively.

[0094] Thus, in this embodiment, by synchronously sampling the source domain perspective sample image and its pixel-level category annotations according to the continuous sampling coordinates, a panoramic distortion view can be obtained. Synchronous distortion annotation The panoramic distortion view corresponding to the source domain perspective sample image can be referenced. Figure 4 The example shown.

[0095] The process described in this embodiment employs inverse mapping starting from the equidistant cylindrical projection output pixels, rather than forward scattering the source domain pixels onto the output canvas. Therefore, each retained output pixel corresponds to a defined sampling position of the source domain perspective sample image, while areas outside the field of view remain invalid.

[0096] See Figure 3 In this embodiment, when constructing the source domain visual cue library, the object focus region and the context region containing the neighborhood background can be generated based on the object annotation region defined by the pixel-level category annotation of the source domain perspective sample image. A panoramic distortion transformation matching the aforementioned panoramic distortion transformation is then applied to at least a portion of the image regions. The object focus region may correspond to the aforementioned first sub-image, and the context region containing the neighborhood background may correspond to the aforementioned second sub-image.

[0097] The image encoder in the open vocabulary masking segmentation model is used to extract the regional visual features of each region image after panoramic distortion transformation, and the regional visual features can be normally normalized. According to the source domain category set, the visual features of each region and their corresponding metadata are written into the corresponding visual cue slot of the source domain visual cue library. The metadata includes, but is not limited to, valid tags, cropping type, source view type, distortion parameters and source side quality information.

[0098] In some embodiments, the image encoder is kept frozen during training. The source domain visual cue library is built offline and kept frozen during the training phase, and it stores one or more visual cue slots corresponding to each source domain category, each visual cue slot containing at least the region visual features corresponding to the source domain perspective sample image.

[0099] In this embodiment, the metadata is used to describe one or more of the following: the source, attributes, validity, and quality of the corresponding visual cue slot. Specifically:

[0100] Valid markers are used to indicate whether the corresponding visual cue slot contains valid regional visual features that can participate in subsequent retrieval and / or aggregation.

[0101] In one implementation, a preset number (e.g., 8) of visual cue slots can be pre-configured for each source domain category. When constructing the source domain visual cue library, if the number of sub-images corresponding to the source domain category exceeds the preset number, the optimal preset number of sub-images are selected from the sub-images corresponding to the source domain category, and the region visual features are extracted and written into the corresponding visual cue slots. If the number of sub-images corresponding to the source domain category is less than the preset number, for example, if the source domain category corresponds to only 6 sub-images, then 6 visual cue slots are written into the corresponding region visual features and marked as valid, while the remaining 2 visual cue slots are set as invalid due to lack of data. Thus, the validity of each visual cue slot can be determined by the valid marking. In subsequent candidate region visual cue retrieval, cue grouping, weighted aggregation, and / or classification score calculation, only the valid visual cue slots indicated by the valid marking are used; empty or invalid slots do not participate in the corresponding calculations. For example, a valid label can be 1 to indicate validity and 0 to indicate invalidity; when a source domain category does not have a valid visual cue slot, that source domain category may not participate in subsequent cross-vocabulary retrieval, thereby avoiding a denominator of zero or adding invalid visual evidence to the candidate set.

[0102] The cropping type indicates the subimage cropping method upon which the visual features of the corresponding region are based. For example, the cropping type can indicate whether the visual features of the region come from the first subimage or the second subimage, i.e., from the object focus area or from a context area that includes the surrounding background. When grouping and aggregating candidate region visual cues later, the candidate region visual cues can be divided into different cue groups such as object focus groups and context groups based on the cropping type.

[0103] The source view type indicates the type of source view upon which the visual features of the corresponding region are based. For example, the source view type can indicate whether the visual features of the region are obtained directly from the original perspective view in the source domain perspective sample image, or from a panoramic distortion view obtained by transforming the source domain perspective sample image into a panoramic distortion view. By recording the source view type, it is easier to distinguish between original view visual evidence and panoramic distortion view visual evidence during the training and / or inference phases, and to match the geometric distribution of the cue library views with the segmentation training views.

[0104] Distortion parameters indicate the transformation parameters used when generating the corresponding panoramic distortion view. For example, distortion parameters may include one or more of the following: camera latitude, horizontal field of view, and effective area cropping parameters. They may also include parameters used to reproduce the panoramic distortion transformation, such as yaw angle and random seed. By recording distortion parameters, the source of geometric deformation of the corresponding region's visual features can be determined, facilitating the reproduction or adjustment of the panoramic distortion transformation when needed.

[0105] Source-side quality information is used to characterize the quality or reliability of visual cues for a corresponding region on the source domain side. For example, source-side quality information may include one or more of the following: effective pixel percentage, target region area percentage, cropped region integrity, boundary sharpness, resampling count, whether the effective pixel percentage threshold is met, and source-side quality score. When the number of sub-images corresponding to a source domain category exceeds a preset number, the optimal sub-image can be selected to construct a visual cue slot by combining source-side quality information. Furthermore, source-side quality information can be used to determine the priority of candidate region visual cues during subsequent cue slot selection or sorting.

[0106] Through the aforementioned metadata, each visual cue slot in the source domain visual cue library not only contains regional visual features, but also carries descriptive information such as effective labels, cropping type, source view type, distortion parameters, and source-side quality information. This facilitates subsequent cross-lexicon retrieval, conditional aggregation, compatibility gating, and text-visual collaborative classification based on target category text features during the target panorama reasoning stage, and automatically degenerates into a text classification path when visual cues are unavailable or unreliable.

[0107] In some embodiments, after writing the visual features and metadata of each region into the corresponding visual cue slots of the source domain visual cue library, the visual cue slots under the same source domain category can be filtered and sorted. This filtering and sorting can be performed offline during the training phase, and the filtered source domain visual cue library remains frozen during subsequent training and inference phases.

[0108] Specifically, for each visual cue slot within the same source domain category, its regional visual features can be used as candidate visual cue pieces to be screened. A weighted combination of the consistency between the candidate visual cue and the text features of the source domain category, and the consistency between the candidate visual cue and the overall visual features of the source domain category, is used to obtain a comprehensive score for the candidate visual cue. The source domain category text features can be obtained by encoding the source domain category names in the source domain category set using a frozen text encoder; the overall visual features of the source domain category can be obtained by aggregating the regional visual features of all valid visual cue slots within that source domain category, for example, by averaging the normalized regional visual features of all valid visual cue slots according to the aforementioned valid labels and then normalizing them again.

[0109] After obtaining the comprehensive score of each candidate visual cue, the candidate visual cue slots within the same source domain category are sorted from highest to lowest comprehensive score, and a preset number of visual cue slots at the top of the ranking are retained. For example, a preset number (e.g., 8) of visual cue slots can be pre-configured for each source domain category. If the number of valid visual cue slots within a source domain category exceeds the preset number, the preset number of visual cue slots with the highest comprehensive scores are retained, and the remaining visual cue slots can be marked as invalid or not added to the source domain visual cue library; if the number of valid visual cue slots within a source domain category is less than the preset number, the insufficient slots can be set as invalid and indicated by the aforementioned valid marking.

[0110] The weights of text consistency and visual consistency can differ depending on the type of source domain category. Preferably, for object categories, the weight of text consistency is 0.70 and the weight of visual consistency is 0.30; for background or material categories, the weight of text consistency is 0.45 and the weight of visual consistency is 0.55. The object, background, or material categories can be pre-defined according to the category definition of the source domain category set. The specific weight values ​​described above are merely preferred examples of this embodiment, and those skilled in the art can adjust them according to the source domain category type, data distribution, and encoder type.

[0111] In one specific implementation, this embodiment employs a frozen CLIP ViT-B / 16 image encoder, retaining no more than eight 512-dimensional normalized visual cue slots for each of the 171 COCOStuff source categories. The serialized source domain visual cue library, including the aforementioned metadata, is approximately 3.48 MB. Through the above filtering and sorting, the visual cue slots in the source domain visual cue library maintain both semantic consistency with the text features of the source domain category and visual consistency with the overall visual features of the source domain category. Subsequently, during the model inference stage, multiple candidate region visual cues can be retrieved from the source domain visual cue library based on the target category text features. Compatibility gating values ​​and collaborative visual features are generated based on the retrieved candidate region visual cues, thereby introducing bounded visual residuals while maintaining the main path of text classification.

[0112] During the model inference stage, candidate region visual cues can be retrieved across the vocabulary from the source domain visual cue library based on the target category names in the target category name set, and then collaborative visual features and compatible gating values ​​can be generated.

[0113] This embodiment retrieves one or more source domain categories based on cross-modal similarity for the text features of the target category, and gathers the regional visual cues of the selected source domain categories to form a candidate set; for target categories that do not have a source category with the same name, it allows the retrieval of alias categories, semantically related categories, superordinate categories, subordinate categories, or source domain categories with related appearance and context.

[0114] The process of generating the compatibility gate value includes: calculating the mean of all candidate region visual cues to obtain the overall visual features; calculating the cosine similarity between the overall visual features and the target category text features; and mapping the cosine similarity to a bounded interval through a bounded continuous mapping function to obtain the compatibility gate value.

[0115] When the target category does not have the candidate region visual cue, the compatibility gate value is configured to zero.

[0116] The process of generating collaborative visual features includes: dividing the multiple candidate region visual cues into two cue groups according to the cropping region type of the source domain perspective sample images corresponding to the multiple candidate region visual cues; for each cue group, weighted aggregation of the region visual features of each candidate region visual cues within the group is performed using the similarity between the target category text features and the visual cues of each candidate region within the group as the weight, to obtain the intra-group visual features of each cue group; fusing the intra-group visual features of the two cue groups, and then concatenating the fused visual features with the target category text features and inputting them into a multilayer perceptron to obtain the collaborative visual features.

[0117] Specifically, see Figure 5 For the target category in the target category name set The frozen text encoder will target the category. The name is encoded as a normalized target category text feature. Subsequently, the target category text features were... Compare with the average visual features of each source category in the source domain visual cue library. Calculate the cosine similarity and sort the source categories from highest to lowest similarity, then select the top-ranked categories. The source category is used as the target category. A collection of related source categories. For example, .

[0118] (7)

[0119] in, Indicates the source domain category index. This indicates the index of the visual cue slot under this source category; For the first Individual source category The visual characteristics of the area of ​​each visual cue slot; For the first Individual source category Effective marking of each visual cue slot; This represents the summation of visual cue slots for that source category; To effectively indicate the number of slots; Indicates L2 normalization; For the first Average visual features of each source category.

[0120] Furthermore, from the above Extract all valid visual cue slots from the source categories to form the target category. The set of visual cues for candidate regions:

[0121] (8)

[0122] in, Indicates the target category index; For target category Normalized text features; Indicates cosine similarity; This indicates that the top categories with the highest similarity are selected from all available source categories based on the similarity between the text features of the target category and the average visual features of each source category. Source category; Indicates the target category The most similar A set consisting of source categories; Indicates source category The first in A visual cue slot; Indicates from the previous The candidate set, comprised of all valid visual cue slots gathered from each source category, represents the target category, and the visual features of the region corresponding to each valid visual cue slot within it constitute the target category. Visual cues for candidate regions.

[0123] In other words, this embodiment first identifies several source categories that are visually semantically related to the target category name. Then, it extracts usable visual cues from these source categories as candidate visual evidence for subsequent text-based conditional visual cue aggregation. The target category name does not need to be the same as the retrieved source category name. For example, the target category "sofa" can be retrieved from the source category "long sofa," and "traffic sign," which has no corresponding source category, can be retrieved from related source categories such as "stop sign" and "traffic light." Thus, the target text is used to determine which part of the source domain visual evidence should be read, rather than directly replacing the target text prototype with the source domain visual prototype.

[0124] This embodiment obtains the target category. The corresponding candidate region visual cue set Next, first, the set The set is obtained by averaging and normalizing the visual features of all regions within the set. Corresponding overall visual features :

[0125] (9)

[0126] in, This indicates the number of valid visual cue slots in the set.

[0127] Furthermore, the candidate region visual cues are divided into an object-focusing group and a context group. The object-focusing group mainly provides the appearance and shape information of the target itself, while the context group provides additional scene information surrounding the target. Within each effective group, the aggregation weight is determined based on the similarity between the target category text features and the visual cue features of each region, and the visual cues within the group are weighted and summarized. Subsequently, the summarized visual features obtained from the two groups are fused in a learnable manner; when one group does not have effective region visual cues, only the other group is used. Finally, the fused summarized visual features are concatenated with the target category text features and input into a multilayer perceptron, and after L2 normalization, the co-visual features are obtained. .

[0128] Next, based on the text features of the target category With overall visual features similarity And obtain compatible gate values ​​through bounded continuous mapping. :

[0129] (10)

[0130] in, For target category Text features With overall visual features Cosine similarity between them Indicates cosine similarity; The larger the value, the better the overall visual evidence provided by the candidate region visual cues matches the target category text; the corresponding compatibility gate value... The larger.

[0131] This represents the Sigmoid function, which sets the compatibility gate value between 0 and 1. The midpoint of the gating system is used to determine at what level of compatibility visual evidence should be introduced more clearly. This is a gated temperature used to control how quickly the compatibility gate value changes with compatibility. For example... , .

[0132] When target category If no valid candidate region visual cue exists, do not use and and order Compatible gating values This will be used to modulate collaborative visual features in subsequent text-visual collaborative classification. Contribution to the classification score.

[0133] In some embodiments, after obtaining collaborative visual features and compatible gating values Next, text-visual collaborative classification can be performed on each candidate mask. Let the normalized mask query feature output by the mask decoder be... The preset residual coefficient is Then the first Each candidate mask corresponds to the target category. The text visual collaborative classification score is:

[0134] (11)

[0135] in, For the first Each candidate mask is for the target category. Text visual collaborative classification score; For the first Transpose of a mask query feature; Text features for the target category; For collaborative visual features;

[0136] superscript Indicates transpose, therefore and Both are vector dot products, which are equivalent to cosine similarity under the unit norm condition; For example, to preset the visual residual coefficient, .

[0137] In the above formula (11), the first term The first item is the text classification item, which constitutes the main path for text classification; the second item... For compatible gating values Modulated visual residual terms. Thus, this embodiment can introduce visual evidence from the source domain region in a bounded visual residual manner while preserving the main path of text open vocabulary recognition, thereby obtaining the text visual co-classification score of the candidate mask relative to each target category.

[0138] After obtaining each candidate mask and its corresponding text-visual co-classification score, class assignment can be performed based on the candidate mask and its corresponding text-visual co-classification score to generate the class segmentation result of the isometric cylindrical projection panoramic image to be segmented. For example, the classification probability of each candidate mask can be fused with the candidate mask probability to output a pixel-level class label map or class probability map corresponding to the input panoramic image.

[0139] like Figure 6 As shown, based on the aforementioned panoramic distortion transformation and / or the construction of the source domain visual cue library, the open vocabulary masking segmentation model can be further trained. The training process of the open vocabulary masking segmentation model includes: acquiring source domain perspective sample images and their corresponding panoramic distortion views, and inputting the source domain perspective sample images and the panoramic distortion views into the open vocabulary masking segmentation model in the form of matching view pairs for training. The source domain perspective sample images and the panoramic distortion views can be obtained through the aforementioned panoramic distortion transformation; their content corresponds, but their projection forms are different. During training, the usage ratio of the panoramic distortion views in the matching view pairs is adjusted according to a curriculum-based schedule, so that the open vocabulary masking segmentation model gradually adapts from the appearance of the source domain perspective sample images to the visual distribution of the panoramic distortion views.

[0140] Specifically, to reduce the impact of strong distortion in the early stages of training on category learning, a curriculum-based scheduling control is used to manage the proportion of the panoramic distorted view in the matching slots of the matching view pairs. Preferably, at the 0th, 10000th, 30000th, 50000th, and 60000th training iterations, the panoramic distortion view ratio is set to 0.25, 0.35, 0.70, 1.00, and 1.00, respectively. For paired slots where the panoramic distortion view is not enabled, the corresponding source domain perspective sample image can be copied as the input view, allowing the open vocabulary masking segmentation model to gradually transition from the easily recognizable appearance of the source domain to the complete panoramic view. Through the above-described curriculum-based scheduling, the stability of category learning can be maintained in the early stages of training, and the panoramic distortion view can be fully utilized in the later stages of training, improving the adaptability of the open vocabulary masking segmentation model to panoramic projection distortion.

[0141] In some embodiments, the source domain perspective sample image and the panoramic distortion view in the matched view pair may each have supervised annotations. For example, the source domain perspective sample image may correspond to its pixel-level category annotation. Panoramic distortion view corresponding to synchronous distortion annotation During training, the open-vocabulary mask segmentation supervision loss can be calculated based on the matched view pairs and their supervision annotations. Optionally, it can be combined with one or more of the structural consistency loss, cue consistency loss, and semantic consistency loss from subsequent distortion consistency cue adaptation to jointly optimize the open-vocabulary mask segmentation model. The curriculum-based scheduling is only used to adjust the proportion of panoramic distortion views used in the matched view pairs and does not change the content correspondence between the source domain perspective sample images and their supervision annotations.

[0142] Specifically, structural consistency is used to constrain the consistency of the structural signatures of the prediction masks of the source domain perspective sample image and the panoramic distortion view, so as to maintain the stability of the overall occupancy and contour changes of the main prediction regions of the two views. Cue consistency is used to constrain the consistency between the positive query features of the same source domain category in the source domain perspective sample image and the panoramic distortion view, as well as the consistency between the co-visual features and the source domain category text features and the overall visual features, respectively, so as to keep the query features of the same category close between the two views and to make the co-visual features simultaneously constrained by textual semantics and candidate visual evidence; here, the source domain category text features are obtained by encoding the source domain category names in the source domain category set using a frozen text encoder. Semantic consistency is used to constrain the bidirectional relative entropy between the category probability distributions of the source domain perspective sample image and the panoramic distortion view, so that the overall judgments of the two views on each category remain close.

[0143] In some embodiments, when calculating the structural consistency loss term, structural signatures are extracted from the predicted soft masks of the original view and the distorted view, respectively. The structural signature is a structural feature vector formed by summarizing one or more soft masks whose prediction confidence and region area meet preset conditions, based on statistical information such as foreground occupancy, spatial variation, edge variation, and horizontal and vertical distribution contours. It is used to summarize the overall shape and spatial distribution of the predicted results for the corresponding views, without requiring the two views to correspond perfectly at pixel positions. Structural consistency loss Represented as:

[0144] (12)

[0145] in A set of paired source domain views and panoramic distortion views formed for content from the same source domain. For the number of pairs, and These represent the source domain view and the panoramic distortion view, respectively. and These are the structural signatures for the two views, used to characterize the overall occupancy, spatial variation, edge variation, and directional distribution of the predicted region. To smooth the L1 distance, used to measure the difference between the structural signatures of two views; The weight of the structural signature consistency item; Auxiliary loss for boundaries and small areas; This represents the weight of the auxiliary item.

[0146] Among them, auxiliary losses in boundary and small areas The prediction mask and its corresponding ground truth mask are calculated from the positive samples matched during source domain training. First, the ground truth mask is dilated and eroded, and the target boundary is determined based on the difference between the two. Second, based on the area ratio of the ground truth mask and the area ratio retained after erosion, target regions with small areas or elongated shapes are identified. Subsequently, the pixel weights of the aforementioned boundary regions and small / elongated regions are increased, and a weighted binary cross-entropy is calculated between the prediction mask and the ground truth mask. The result is used as the target boundary. This auxiliary loss does not depend on specific category names and is used to avoid the conventional masking loss being dominated by large areas and ignoring target boundaries and fine structures.

[0147] In formula (12), the first term compares whether the overall mask structure of the source domain view and the panoramic distortion view is similar, and the second term applies additional supervision to the target boundary and small areas, thereby avoiding the model from only maintaining the overall statistics of large areas and ignoring boundary details.

[0148] In some embodiments, positive queries are selected using the Hungarian matching results of the segmentation head during source domain training. The positive queries of the same assigned category in both the source domain view and the panoramic distortion view are averaged and L2 normalized to obtain the mean of the category queries for the two views. Simultaneously, the co-visual features are constrained to approximate the frozen text features and the mean of the candidate visual features. Specifically, a consistency loss is suggested. Represented as:

[0149] (13)

[0150] in, For the positive query triples of the category determined by the Hungarian matching results gather, Its quantity; and Categories in the source domain view and the panoramic distortion view, respectively. The positive query mean and features after L2 normalization; Weights for pairwise query consistency items; The set of categories that participate in cue anchoring and have valid visual evidence. Its number of categories; This indicates the weight of the anchor term.

[0151] In formula (13), the first term keeps the query features of the same category close in both views, and the second term makes the collaborative visual features close to the corresponding text features and candidate visual evidence at the same time, thereby avoiding the fusion result from deviating from the semantics of the target category.

[0152] In some embodiments, the foreground category probabilities are aggregated into source domain view category distribution and panoramic distortion view category distribution in a query order-independent manner, and the semantic output of the two views is constrained by bidirectional relative entropy:

[0153] (14)

[0154] in, This represents the semantic consistency loss. and These are the foreground category probability distributions obtained by summarizing the source domain view and the panoramic distortion view in a query order-independent manner; This represents the Kullback-Leibler relative entropy. This indicates that the relative entropy is calculated using the latter distribution as a reference distribution; the KL terms in both directions together constitute a bidirectional semantic consistency constraint; coefficients Used to average the views in both directions and all matching view pairs.

[0155] The semantic consistency loss in this embodiment does not require a one-to-one correspondence for each query, but rather compares the category probability distributions after the two views are summarized. This embodiment uses relative entropy in two directions for constraint, which can more symmetrically measure the difference between the semantic judgments of the two views.

[0156] Based on the loss term provided in the above embodiments, the loss function used in the training process of this model is:

[0157] (15)

[0158] in, , , These are the weighting coefficients for the corresponding loss terms.

[0159] After training the model using the above embodiments, the trained model can be used for inference. For example... Figure 7As shown, during model inference: the equidistant cylindrical projection panoramic image to be segmented and a predefined set of target category names are obtained and input into the trained open vocabulary mask segmentation model. The model extracts features from the isometric cylindrical projection panoramic image to be segmented, generating at least one candidate mask and its corresponding mask query feature; it text-encodes the names of each target category to generate target category text features; it retrieves multiple candidate region visual cues matching the target category text features from a frozen source domain visual cue library across the vocabulary; it generates a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features, and fuses the multiple candidate region visual cues according to the target category text features to generate collaborative visual features; for each candidate mask, based on the similarity between its mask query feature and the target category text features and the collaborative visual features, it determines a text classification term and a visual residual term modulated by the compatibility gate value; it obtains the text visual collaborative classification score of the candidate mask relative to each target category based on the text classification term and the visual residual term; and it assigns categories based on each candidate mask and its corresponding text visual collaborative classification score to generate the category segmentation result of the isometric cylindrical projection panoramic image to be segmented.

[0160] During the inference phase, multi-dimensional panoramic distortion transformation and distortion consistency cue adaptation are not performed. Target panoramic image annotation and target pseudo-labels are not used, and the source domain visual cue library is not updated based on the target panoramic image. Therefore, while maintaining the main path of open-vocabulary text recognition, bounded corrections to the classification results are performed using regional visual evidence from a fixed source domain visual cue library. This allows for open-vocabulary panoramic image segmentation without using target domain training data or updating the visual cue library based on the target image.

[0161] To verify the feasibility and technical effectiveness of the present invention, this embodiment uses COCOStuff-164k as the source domain training data and a fixed source domain visual cue library construction data under a strict source domain training protocol, and tests are conducted on three open-vocabulary panoramic segmentation benchmarks: Stanford2D3D, DensePASS, and Matterport3D. Target panoramic images, target annotations, target pseudo-labels, and cuees generated from target images are not included in the training or cue library construction.

[0162] In this embodiment, mean Intersection over Union (mIoU) is used as the main segmentation evaluation metric, where Intersection over Union represents the ratio of the intersection area to the union area between the predicted region and the ground-truth annotated region, and mIoU is the average value of the Intersection over Union of each evaluation category. A larger value indicates better segmentation performance. In addition, this embodiment uses WordNet 3.0 as the basis for category semantic mapping. WordNet 3.0 is a lexical semantic database that can be used to distinguish different meanings corresponding to the same word and establish semantic correspondence between different category names. In this embodiment, the semantic mapping corresponding to the target category is predetermined and frozen, which is only used for the evaluation calculation of the Open category mean Intersection over Union (Open mIoU), and does not participate in model training, source category retrieval or visual residual gating.

[0163] Table 1: Dataset and evaluation settings

[0164]

[0165] It can be seen from Table 1 that the source domain training data adopts normal field-of-view images, while all three test benchmarks are panoramic images, and the target category space is not completely consistent with the COCOStuff source category space. Therefore, the experiment includes both geometric transfer from normal field-of-view images to panoramic images and open-vocabulary semantic transfer from source categories to target categories.

[0166] Table 2: Main implementation parameters

[0167]

[0168] Based on the above parameters in this embodiment, polynomial learning rate decay is adopted for training, the text encoder and the image encoder for prompt library construction remain frozen, and the checkpoints are selected according to the mIoU of the source domain verification set. The target vocabulary prompt representation can be cached in advance, and both ML-PDS and DCPA are executed only in the training phase.

[0169] Table 3: Overall segmentation results (unit: %)

[0170]

[0171] OOOPS is an open panoramic segmentation model proposed in the document "Open Panoramic Segmentation". The full English name of its corresponding task is "Open Panoramic Segmentation", and the Chinese name is "开放全景分割"; OOOPS is the proprietary model name adopted in the original text, and no separate letter-by-letter expansion is given in the original text. This method adopts a frozen vision-language model and a deformable adapter network, and enhances the model's adaptability to panoramic distortion through random equirectangular projection.

[0172] ACD4OPS stands for "Augmenting and Contrasting Distortion for Open Panoramic Segmentation". This method generates distorted training images through random Gaussian deformation enhancement and performs patch-level and pixel-level contrastive learning between the original and distorted images to improve the model's distortion perception capability and feature stability.

[0173] PanoTVP stands for "Panorama-Aware Text-Visual Prompt Collaboration". The method, titled "A Collaborative Approach for Open Panoramic Segmentation: Text-Visual Cueing," utilizes target category text retrieval to obtain visual cues from a fixed source domain. It supplements the text classification results with bounded visual residuals gated by compatibility, and enhances the model's stability on panoramic images through multi-dimensional panoramic distortion synthesis and distortion consistency cue adaptation. ACD4OPS* in the table indicates that the corresponding experimental results are cited from the original paper on this method.

[0174] The specific implementation schemes of the three methods mentioned in Table 3 can be found in relevant technologies, and will not be repeated here.

[0175] As shown in Table 3, the mIoU of this invention on Stanford2D3D, DensePASS, and Matterport3D reached 46.67%, 50.25%, and 32.92%, respectively, with an average mIoU of 43.28%. Compared to the OOOPS baseline, the results on the three datasets were improved by 2.17, 3.25, and 1.72 percentage points, respectively; and compared to the publicly reported ACD4OPS results, the results were improved by 1.17, 1.15, and 1.02 percentage points, respectively. The Open mIoU of PanoTVP on the three datasets were 50.88%, 55.94%, and 39.13%, respectively, with an average of 48.65%. ACD4OPS* represents the value reported in the published paper.

[0176] Table 4: Cumulative ablation results of components (mIoU, unit: %)

[0177]

[0178] Table 4 shows that when only the text-visual cue collaboration module is added, the average mIoU increases from 40.90% to 41.59%; after further introducing the multi-dimensional panoramic distortion synthesis component, it increases to 42.21%; and after fully adding the distortion consistency cue adaptation component, it reaches 43.28%. These results demonstrate the effectiveness of target text-driven visual evidence transfer, the coupling of segmentation training with the cue library's geometric view, and cross-view... Figure 1 Consistency constraints have a complementary effect.

[0179] Table 5: Audit Grouping Results of Target Category and Search Source Category (Unit: %)

[0180]

[0181] Table 5 presents statistics for the 52 target dataset entries based on the current experimental audit labels. The related and open groups showed an average improvement of 2.30 and 3.73 percentage points in OOOPS, respectively, indicating that even when the target category vocabulary is not entirely consistent with the source category vocabulary, the target category text can still retrieve source-side visual evidence with usable appearance or contextual information. The terms "same name," "alias," "related," and "open" are used only as experimental audit labels in this embodiment and do not participate in model training, prompt retrieval, gating calculation, or inference, nor do they constitute a general or complete classification rule for semantic relationships of any category.

[0182] Table 6: Coupling of Multi-Dimensional Panoramic Distortion Composite Views and Robustness of Panoramic Regions (mIoU, Unit: %)

[0183]

[0184] In Table 6, Standard represents the IoU of the entire image; Equat., Mid, and Polar represent the latitudinal regions where |ℓ|≤30 degrees, 30 degrees<|ℓ|≤60 degrees, and |ℓ|>60 degrees, respectively; Seam represents the stitched region with a width of 5% on the outermost left and right sides of the image; and Sph. represents the spherical index weighted by the cosine of latitude. The results show that using ML-PDS simultaneously for both the training view and the cue library view on all three datasets achieves an overall improvement relative to the single-dimensional ERP baseline, indicating that using a panoramic deformation view that matches the training branch with the visual cue evidence has practical effects.

[0185] Table 7: DCPA Consistency Loss Grouping Experiment (mIoU, unit: %)

[0186]

[0187] As shown in Table 7, the complete distortion consistency cue adaptation configuration has the highest average value across the three datasets, and achieves the strongest results on Stanford2D3D and DensePASS; the intermediate configuration for cue consistency and semantic consistency achieves high results on Matterport3D. This experiment demonstrates that the three types of consistency losses need to be understood as collaborative constraints, rather than simply interpreted as each additional loss term monotonically increasing the performance of each dataset.

[0188] Table 8: Training and Deployment Computational Costs

[0189]

[0190] As shown in Table 8, this invention increases the number of parameters by 3.15M compared to OOOPS, accounting for approximately 2.0%. The fixed source domain visual cue library is approximately 3.48 MB, not included in the model parameter count. The four-card training time increases by approximately 15.6%, and the single-card inference latency increases by approximately 9.3%. The multi-dimensional panoramic distortion synthesis component and the distortion consistency cue adaptation component are both training phase mechanisms; target domain optimization is not performed during the target panoramic inference phase.

[0191] Table 9: Source-side visual cue retrieval audit for representative target categories

[0192]

[0193] The audit labels in Table 9 are only used to illustrate the relationship and similarity between the target category and the search source category. Both the threshold value g and the threshold value g are calculated using a frozen target category text and source domain visual cue library before processing the target image. The candidate source categories and their visual cues do not originate from the target panoramic image. The similarity of all 52 target entries is calculated. The similarity coefficient ranges from 0.272 to 0.316, and the gate value g ranges from 0.363 to 0.578; at λ=0.35, the actual visual residual coefficient remains between approximately 0.127 and 0.202, thus always remaining within a bounded range. The audit label is not involved in the calculation of the above similarity, gate value, or visual residual.

[0194] Table 10: Stanford 2D / 3D Complete Category IoU (Unit: %)

[0195]

[0196] Table 10 retains all 13 evaluation categories of Stanford 2D3D. Compared to OOOPS, this invention shows significant improvements in categories such as "panel," "ceiling," "miscellaneous items," and "wall," while "door" and "table" show declines. This result indicates that the benefit of source domain visual cues depends on the degree of matching between the source-side visual evidence and the actual appearance of the target category. Therefore, compatibility gating is used to continuously constrain visual residuals, rather than completely replacing the text prototype with the visual prototype.

[0197] Table 11: DensePASS Complete Category IoU (Unit: %)

[0198]

[0199] Table 11 retains all 19 evaluation categories of DensePASS. "Traffic lights", "traffic signs", "buses", "walls", and "vegetation" showed significant improvements, indicating that the target category text can retrieve source-side visual evidence with appearance or contextual relevance even when the source category vocabulary is not completely consistent. A few categories showed slight declines, reflecting that there may still be residual mismatches in the cue library when the appearance of the target domain changes.

[0200] Table 12: Matterport3D Complete Category IoU (Unit: %)

[0201]

[0202] Table 12 retains all 20 evaluation categories of Matterport3D. This invention achieves improvements in categories such as "table," "toilet," "shelf," "mirror," "shower area," and "bathtub," with "shower area" showing a 7.52 percentage point improvement. Categories such as "wall," "sink," and "stairs" show declines, further illustrating that source category coverage does not necessarily mean a perfect match in target appearance; bounded gating residuals can limit the impact of mismatched visual evidence on the open lexical path of the text.

[0203] Table 13: Open mIoU results under unified frozen WordNet 3.0 mapping (unit: %)

[0204]

[0205] The OOOPS results in Table 13 were recalculated using the same frozen WordNet 3.0 mapping and the same target dataset evaluation settings as PanoTVP. The Stanford2D3D three-fold Open mIoU were 51.3699%, 44.4728%, and 50.1341%, respectively, with an arithmetic mean of 48.65895% (recorded as 48.66% to two decimal places in the table). The DensePASS result was 54.06%, and the Matterport3D result was 37.51%, with an average of 46.74% across the three datasets. The corresponding PanoTVP results were 50.88%, 55.94%, and 39.13%, with an average of 48.65%. Taking Stanford2D3D as an example, the frozen mapping used fixed WordNet 3.0 word meanings for "beam" and "panel," such as beam.n.02 and board.n.03, respectively. The above mappings and Open mIoU are used only for evaluation and are not involved in training, source category retrieval, or visual residual gating.

[0206] Based on the above embodiments of the present invention, the open vocabulary panoramic image segmentation method of the present invention has achieved at least the following beneficial effects:

[0207] (1) This invention addresses both the geometric domain differences caused by panoramic projection and the semantic differences caused by the open target vocabulary. The sample view and the source domain visual cue library adopt a matching multi-dimensional panoramic distortion transform family, which can reduce the geometric distribution mismatch between ordinary field-of-view visual cues and panoramic mask queries.

[0208] (2) The present invention uses information such as local appearance and context in the pixel-level annotation area of ​​the source domain to supplement the target category name text information, and the target category text determines the retrieval range of visual evidence in the source domain. Therefore, the target category does not need to be completely consistent with the source domain category name, and can support the transfer of visual evidence between aliases, semantically related categories and different category granularities.

[0209] (3) This invention retains the basic classification score between the target category text and the mask query, and only adds source domain visual evidence as a bounded residual with compatible gating to the classification result. When the visual evidence is unreliable or does not exist, the residual is attenuated or turned off, and the classification path can degenerate into pure text recognition, thereby taking into account both the utilization of source domain visual evidence and the ability to transfer open vocabulary.

[0210] (4) This invention, through joint training of structural consistency, prompt consistency and semantic consistency, makes the source domain view and the distorted view stable in terms of mask structure, query-prompt relationship and final semantic distribution, which can improve the robustness of the model to latitude-related stretching, panoramic stitching area and target shape changes.

[0211] (5) The source domain visual cue library of the present invention is constructed from source domain images and source domain annotations. When performing target panoramic reasoning, there is no need for panoramic sample training, panoramic target pseudo-labels, or updating of the cue library according to panoramic target images. Experimental results show that the average mIoU of the complete scheme on the three panoramic benchmarks of Stanford2D3D, DensePASS and Matterport3D is increased from 40.90% to 43.28% of the baseline, and gains are achieved in multiple categories and different panoramic regions.

[0212] Figure 8 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 8 At the hardware level, the device includes a processor 810, an internal bus 820, a network interface 830, memory 840, a hardware acceleration device 850, and non-volatile memory 860, and may also include other hardware required for its functions. One or more embodiments of the present invention can be implemented in software, for example, the processor 810 reads the corresponding computer program from the non-volatile memory 860 into the memory 840 and then runs it. Of course, in addition to software implementation, one or more embodiments of the present invention do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0213] This invention also provides an open vocabulary panoramic image segmentation device based on visual-language cues collaboration, which can be applied to, for example... Figure 8 The present invention is implemented in the electronic device shown. Figure 9 This is a block diagram illustrating an open vocabulary panoramic image segmentation device based on visual-language cues collaboration, as shown in an exemplary embodiment of the present invention. Figure 9 As shown, the open vocabulary panoramic image segmentation device includes:

[0214] The data acquisition unit 910 is used to acquire the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names and input them into the trained open vocabulary mask segmentation model.

[0215] The data processing unit 920 is used to generate the category segmentation result of the panoramic image through the model; wherein, the open vocabulary mask segmentation model performs the category segmentation task of the previous related embodiments, and the specific process can be referred to the previous related embodiments, which will not be repeated here.

[0216] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0217] Accordingly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0218] Accordingly, embodiments of the present invention also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0219] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0220] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0221] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0222] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0223] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0224] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0225] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0226] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0227] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An open-vocabulary panoramic image segmentation method based on visual-linguistic cues, characterized in that, The method includes: Obtain the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names, and input them into a trained open-vocabulary mask segmentation model. The model then generates the category segmentation result for the panoramic image. The open-vocabulary mask segmentation model performs the following steps: Step S1: Extract features from the panoramic image to generate at least one candidate mask and its corresponding mask query feature; Step S2: Text encoding is performed on each of the target category names to generate target category text features; Step S3: Query multiple candidate region visual cues that match the text features of the target category from the source domain visual cue library; Step S4: Generate a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features; and fuse the multiple candidate region visual cues based on the target category text features to generate collaborative visual features. Step S5: For each candidate mask, the text visual co-classification score of the candidate mask relative to each target category is obtained based on its mask query features, target category text features, co-visual features and compatibility gate value. Step S6: Assign categories based on each candidate mask and its corresponding text visual co-classification score to generate the category segmentation result.

2. The method according to claim 1, characterized in that, The source domain visual cue library in step S3 is built offline and kept frozen during the training phase. It stores one or more visual cue slots corresponding to each source domain category. Each visual cue slot contains at least the regional visual features corresponding to the source domain perspective sample image. The construction process of the source domain visual cue library includes: Based on each source domain category in the source domain category set, for the object annotation region in the source domain perspective sample image containing the object of that source domain category, a first sub-image and a second sub-image are cropped respectively. The first sub-image includes the object region, and the second sub-image includes the neighboring region of the object. Perform panoramic distortion transformation on the first sub-image and the second sub-image respectively to obtain panoramic distortion images corresponding to each sub-image; The image encoder in the open vocabulary masking segmentation model is used to extract the regional visual features of the panoramic distortion map of each sub-image; According to the source domain category set, the visual features of each region and its corresponding metadata are written into the corresponding visual cue slot of the source domain visual cue library.

3. The method according to claim 2, characterized in that, The step of performing panoramic distortion transformation on the first sub-image and the second sub-image respectively includes: Obtain transformation parameters, including camera latitude, horizontal field of view, and effective region clipping parameters; On the equidistant cylindrical projection work canvas, an inverse mapping relationship is constructed from the pixels of the panoramic distortion image to the pixels of the first sub-image or the second sub-image according to the transformation parameters. According to the inverse mapping relationship, the first sub-image and the second sub-image are resampled and interpolated respectively to obtain the corresponding panoramic distortion image.

4. The method according to claim 1, characterized in that, Step S4 generates a compatible gate value in the following manner: Calculate the mean of all candidate region visual cues to obtain the overall visual features; Calculate the cosine similarity between the overall visual features and the target category text features; The cosine similarity is mapped to a bounded interval through a bounded continuous mapping function to obtain the compatibility gate value.

5. The method according to claim 1, characterized in that: When the target category does not have the candidate region visual cue, the compatibility gate value is configured to zero.

6. The method according to claim 1, characterized in that, Step S4 generates collaborative visual features in the following manner: Based on the cropping region type of the source domain perspective sample image corresponding to the multiple candidate region visual cues, the multiple candidate region visual cues are divided into two cues groups; For each of the aforementioned prompt groups, the regional visual features of each candidate region visual prompt within the group are weighted and aggregated using the similarity between the target category text features and the visual prompts of each candidate region within the group as the weight, to obtain the intra-group visual features of each of the aforementioned prompt groups. The visual features within the two cue groups are fused, and the fused visual features are concatenated with the target category text features and then input into a multilayer perceptron to obtain the collaborative visual features.

7. The method according to claim 1, characterized in that, Step S5 includes: Based on the first similarity between the mask query features of each candidate mask and the text features of the target category, a text classification item is obtained; Based on the second similarity between the mask query features of each candidate mask and the collaborative visual features, and modulating the second similarity with the compatibility gate value, a visual residual term is obtained; The candidate mask is obtained as a text visual co-classification score relative to each target category based on the text classification term and the visual residual term.

8. The method according to claim 1, characterized in that, The training process of the open vocabulary mask segmentation model includes: Obtain source domain perspective sample images and their corresponding panoramic distortion views, and train the open vocabulary mask segmentation model by matching the source domain perspective sample images and the panoramic distortion views with the input view. During training, the proportion of panoramic distortion views used in the matching view pairs is adjusted according to a course-based scheduling, so that the open vocabulary mask segmentation model gradually adapts from the appearance of the source domain perspective sample images to the visual distribution of the panoramic distortion views.

9. The method according to claim 8, characterized in that, The loss function used in the training process is: ； in, Supervision loss for open vocabulary mask segmentation; The structural consistency loss is used to constrain the consistency of the predicted mask between the source domain perspective sample image and the panoramic distortion view in terms of structural signature. To indicate consistency loss, constraints are used to determine the consistency between the source domain perspective sample image and the positive query features of the same source domain category matched and assigned in the panoramic distortion view, as well as the consistency between the collaborative visual features and the source domain category text features and the overall visual features, respectively; the source domain category text features are obtained by encoding the source domain category names in the source domain category set used during the model training phase using a frozen text encoder; The semantic consistency loss is used to constrain the bidirectional relative entropy between the category probability distributions of the source domain perspective sample image and the panoramic distortion view; , , These are the weighting coefficients for the corresponding loss terms.

10. An open-vocabulary panoramic image segmentation device based on visual-linguistic cues, characterized in that, The apparatus is used to perform the method as described in any one of claims 1 to 9, the apparatus comprising: The data acquisition unit is used to acquire the isometric cylindrical projection panoramic image to be segmented and a predefined set of target category names, and input them into the trained open vocabulary mask segmentation model. A data processing unit is configured to generate category segmentation results for the panoramic image using the model; wherein the open-vocabulary mask segmentation model performs the following steps: Step S1: Extract features from the panoramic image to generate at least one candidate mask and its corresponding mask query feature; Step S2: Text encoding is performed on each of the target category names to generate target category text features; Step S3: Query multiple candidate region visual cues that match the text features of the target category from the source domain visual cue library; Step S4: Generate a compatibility gate value based on the similarity between the overall visual features of the multiple candidate region visual cues and the target category text features; and fuse the multiple candidate region visual cues based on the target category text features to generate collaborative visual features. Step S5: For each candidate mask, the text visual co-classification score of the candidate mask relative to each target category is obtained based on its mask query features, target category text features, co-visual features and compatibility gate value. Step S6: Assign categories based on each candidate mask and its corresponding text visual co-classification score to generate the category segmentation result.

11. An electronic device, characterized in that, include: processor; as well as A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 9.

Images