A six-face projection-based panoramic image pseudo-label generation method and device

Abstract

The application provides a panoramic image pseudo-label generation method and device based on six projection. The method comprises: performing geometric transformation on an ERP format panoramic image to obtain transmission subgraphs under different viewing angles; performing target detection and semantic understanding on each transmission subgraph and the panoramic image respectively to obtain a target detection frame corresponding to each transmission subgraph, a category and a confidence, and a second pseudo-label set corresponding to the panoramic image; projecting the target detection frame of each transmission subgraph into the coordinate system of the panoramic image and performing screening to obtain a first pseudo-label set; and performing fusion processing on the two pseudo-label sets to obtain pseudo-label data of the panoramic image. The technical scheme can effectively alleviate the influence of geometric distortion of the panoramic image on the detection performance, improve the integrity and accuracy of the pseudo-label through a cross-path fusion mechanism, reduce the dependence on manual labeling, and realize efficient migration of a plane domain open vocabulary detection model to a panoramic image domain.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to a method and apparatus for generating pseudo-tags for panoramic images based on six-view projection. Background Technology

[0002] With the rapid development of technologies such as autonomous driving and intelligent security, 360-degree panoramic cameras have become an important component of environmental perception systems due to their unique advantage of capturing complete environmental information at once, and their applications are becoming increasingly widespread. In such applications, panoramic images are typically stored and represented using the Equirectangular Projection (ERP) format. However, the ERP format introduces significant geometric distortions into panoramic images, especially in the top and bottom regions of the image, where objects undergo severe stretching and deformation, posing a significant technical challenge to visual tasks such as object detection.

[0003] In recent years, Open-Vocabulary Object Detection (OVD) has become a research hotspot in the field of object detection, overcoming the limitations of traditional detection methods that rely on predefined category sets. This technology aims to identify and locate objects in images based on arbitrary text descriptions, offering strong flexibility and scalability. Many related OVD methods are built upon vision-language pre-trained models, such as the detection framework based on Contrastive Language-Image Pre-Training (CLIP), which predicts open categories by calculating the similarity between image region features and text embeddings.

[0004] Currently, mainstream OVD models are primarily trained on planar image datasets and exhibit excellent detection performance on planar images. However, when these models are directly applied to severely distorted ERP panoramic images, their detection capabilities significantly decline. The main reason for this is the significant domain differences between planar and panoramic images, including differences in geometric structure and object appearance distribution, making it difficult for the models to generalize effectively.

[0005] In summary, the relevant technologies lack an efficient and reliable means to generate high-quality pseudo-labels for panoramic image domains where labeled samples are scarce, which in turn restricts the practical deployment and application of open vocabulary detection technology in panoramic scenes. Summary of the Invention

[0006] In view of this, this application proposes a method and apparatus for generating pseudo-labels for panoramic images based on six-view projection. This invention has significant advantages in reducing data annotation costs, improving pseudo-label quality, enhancing model generalization ability, and promoting the practical application of panoramic open vocabulary target detection technology.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] According to a first aspect of the embodiments of this specification, a method for generating pseudo-tags for panoramic images based on six-view projection is provided, comprising the following steps:

[0009] Step S1: Perform geometric transformation on the panoramic image in equidistant cylindrical projection ERP format to obtain transmission sub-images from different viewpoints.

[0010] Step S2: Using a pre-trained open vocabulary detection model, target detection is performed on each transmission sub-image and the panoramic image respectively, to obtain the detection results of each transmission sub-image and the detection results of the panoramic image. The detection results include target detection boxes and detection confidence scores.

[0011] Step S3: Perform semantic understanding on each target detection box to obtain the target category and category confidence corresponding to each target detection box;

[0012] Step S4: Backproject the target detection box of each transmission sub-image to the coordinate system of the panoramic image and filter it to obtain a first pseudo-label set; and obtain a second pseudo-label set based on the detection results and semantic understanding results of the panoramic image.

[0013] Step S5: Perform a fusion process on the first pseudo-label set and the second pseudo-label set, and obtain the pseudo-label data of the panoramic image based on the fusion result.

[0014] According to a second aspect of the embodiments of this specification, a panoramic image pseudo-tag generation device based on six-view projection is provided, comprising:

[0015] The projection transformation unit is used to perform geometric transformations on panoramic images in equidistant cylindrical projection ERP format to obtain perspective sub-images from different viewpoints.

[0016] The object detection unit is used to perform object detection on each perspective sub-image and the panoramic image using a pre-trained open vocabulary detection model, and to obtain the detection results of each perspective sub-image and the detection results of the panoramic image. The detection results include object detection boxes and detection confidence scores.

[0017] The category detection unit is used to perform semantic understanding on each target detection box to obtain the target category and category confidence of each target detection box.

[0018] The label integration unit backprojects the target detection boxes of each perspective sub-image onto the coordinate system of the panoramic image and filters them to obtain a first set of pseudo-labels; and obtains a second set of pseudo-labels based on the detection results and semantic understanding results of the panoramic image.

[0019] The pseudo-label generation unit is used to perform fusion processing on the first pseudo-label set and the second pseudo-label set, and obtain pseudo-label data of the panoramic image based on the fusion result.

[0020] According to a third aspect of the embodiments of this specification, an electronic device is provided, including a processor; and a computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method described in the first aspect.

[0021] The embodiments of this application have at least the following technical effects:

[0022] (1) The embodiments of this application cleverly solve the core bottleneck problem of the lack of panoramic image annotation data. By using a high-performance open vocabulary detection model pre-trained on a large-scale planar dataset, the embodiments of this application can automatically generate a large number of high-quality pseudo-labels for unlabeled panoramic images, greatly reducing the dependence on manual annotation, saving costs, and making it possible to quickly build a large-scale, high-quality panoramic detection dataset, which powerfully promotes the practical application and iterative development of panoramic detection technology.

[0023] (2) In this embodiment of the application, by converting the panoramic image from the severely distorted ERP format into six independent cubic projection surfaces, the negative impact of geometric distortion on the detection model is effectively mitigated. Each cubic surface is closer to a standard planar perspective image, and its geometric characteristics are more matched with the distribution of training images of the open vocabulary detection model, so that the planar model can perform at its best and generate more accurate and reliable initial detection results for each projection surface, laying a solid foundation for subsequent steps.

[0024] (3) Based on the back projection of the six-sided projection pseudo-label, this application introduces a cross-path pseudo-label fusion mechanism to uniformly model and fuse the detection results generated by the cube projection path and the candidate results obtained by direct detection of the ERP image. Since the two types of detection paths have significant structural differences in error distribution, the cube projection path has better geometric consistency in local areas, which can improve the detection capability of small-scale targets and detailed areas, while the ERP direct detection path has advantages in target integrity. Therefore, by introducing a fusion strategy based on cross-union-ratio matching and confidence weighting, the advantages of different path detection results can be complemented. This mechanism can effectively alleviate the target truncation problem across projection planes and reduce the impact of polar distortion on the detection results, thereby significantly improving the overall quality and robustness of pseudo-labels in terms of target integrity, positioning accuracy and small target detection capability.

[0025] (4) The pseudo-label mapping and fusion mechanism proposed in this application has high robustness and accuracy. Through the strict geometric mapping relationship between cube projection and ERP coordinates, and the path-aware weight adjustment strategy introduced in the fusion process, it can effectively deal with the problem of repetition, inconsistency or local deviation between detection boxes from different sources, and finally generate unified, complete and spatially accurate panoramic pseudo-labels. This process avoids the boundary artifacts, target missing or positioning offset problems that may be caused by simple stitching or single path detection, thereby significantly improving the quality of pseudo-labels.

[0026] (5) The panoramic image pseudo-labels of this application embodiment can be used to train the panoramic open vocabulary detection model, so that it has strong generalization ability and practicality. It can not only inherit the open vocabulary recognition ability of the open vocabulary detection model and respond to any text query, but also fully learn the geometric structure characteristics and cross-view semantic consistency in the panoramic image through training or distillation on the fused panoramic pseudo-label data, thereby achieving more stable and accurate target detection performance in real panoramic scenes. In addition, the panoramic image pseudo-labels of this embodiment do not depend on a specific planar open vocabulary detection model architecture, and can be flexibly combined with various visual language models and detection frameworks that are constantly developing now and in the future, with good scalability and foresight. Attached Figure Description

[0027] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Some specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings in an exemplary and non-limiting manner. The same reference numerals in the drawings indicate the same or similar parts or components. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:

[0028] Figure 1 This is a schematic diagram illustrating the overall process of a panoramic image pseudo-tag according to an exemplary embodiment of this application;

[0029] Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of the present application of a method for generating pseudo-tags for panoramic images based on six-view projection;

[0030] Figure 3 This is a structural block diagram of an electronic device illustrated in an exemplary embodiment of this application;

[0031] Figure 4 This is a structural block diagram of a panoramic image pseudo-tag generation device based on six-sided projection, as illustrated in an exemplary embodiment of this application. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0033] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0034] The purpose of this application is to provide a cross-domain pseudo-tag construction scheme based on six-view projection, such as... Figure 1 As shown, by converting the panoramic image into six perspective sub-images of a cube, candidate detection boxes are generated in the planar domain using open vocabulary detection capabilities. Category labels are automatically generated by combining a visual language generation model and a semantic matching mechanism. The results are then back-projected back into the panoramic image coordinate system to construct a set of pseudo-labels. Finally, the pseudo-labels and real labels are combined using a dynamic category expansion mechanism to train the open vocabulary detection model, thereby improving the model's open vocabulary detection capability in panoramic scenes.

[0035] This application's technical solution geometrically transforms the ERP panoramic image, using spherical back-projection mapping to convert it into six perspective sub-images of a cube, corresponding to the front, back, left, right, top, and bottom directions. Each direction uses fixed field-of-view parameters for perspective projection. Since the perspective sub-images no longer exhibit severe lateral distortion, an open vocabulary detection model can be directly applied for object detection bounding boxes. Simultaneously, the ERP panoramic image is directly input into this open vocabulary detection model to obtain object detection bounding boxes in the panoramic domain, thus forming two detection results generated from different paths: one from the cube projection path and the other from the direct ERP detection path.

[0036] For each object detection box, this application's technical solution introduces a visual language generation model to generate a semantic description. For example, perspective sub-images in six perspective directions and the coordinates of the object detection box are input into the BLIP2-Flan-T5-XL model, and a fixed prompt "Name the main thing within the area" is used to generate a text description for the object detection box. Subsequently, the natural language processing tool spaCy is used to perform part-of-speech tagging and noun extraction on the generated text, and the extracted core nouns are used as candidate categories for the object detection box. To improve semantic consistency, the cropped image of the detection box is further input into the CLIP ViT-B / 32 image encoder to extract visual features, while the candidate nouns are input into the CLIP text encoder to extract text features. Both are then subjected to L2 normalization, and cosine similarity is calculated. The noun with the highest similarity is selected as the final pseudo-label category for the detection box, and the similarity score is used as the confidence level of the pseudo-label. Only pseudo-label samples with a confidence level greater than or equal to 0.25 are retained to improve the quality of the pseudo-labels.

[0037] After obtaining the target detection boxes and corresponding categories of the six perspective sub-images, the target detection boxes in each perspective sub-image are back-projected back into the original ERP image coordinate system using inverse spherical mapping. The bounding rectangle corresponding to the set of projection points is then calculated, and this bounding rectangle is used as the back-projected detection box. After completing the above back-projection, the back-projected detection boxes obtained from the cube projection path and the target detection boxes obtained from the direct ERP detection path are uniformly mapped to the same ERP panoramic image coordinate system. A cross-path pseudo-label fusion strategy based on intersection-union matching and confidence weighting is then used to fuse the two types of detection boxes to obtain more complete and robust pseudo-label data.

[0038] This application, through the above-mentioned technical solution, can construct panoramic pseudo-labels by utilizing the mature open vocabulary detection capabilities of the planar domain without manual annotation, thereby achieving cross-domain migration from the planar domain to the panoramic domain. This solution does not change the main structure of the original open vocabulary detection model, has good compatibility and scalability, and can significantly improve the model's open category detection capability in 360° panoramic scenes.

[0039] The embodiments described in this specification will now be described in detail.

[0040] This application provides a method for generating pseudo-tags for panoramic images based on six-view projection. Figure 2 This is a schematic flowchart illustrating an exemplary embodiment of a panoramic image pseudo-tag generation method based on six-view projection, as shown in this application. Figure 2 As shown, the panoramic image pseudo-label generation method includes at least the following steps:

[0041] Step S1: Perform geometric transformation on the panoramic image in equidistant cylindrical projection ERP format to obtain perspective sub-images from different viewpoints.

[0042] Step S2: Using a pre-trained open vocabulary detection model, target detection is performed on each perspective sub-image and the panoramic image respectively, to obtain the detection results of each perspective sub-image and the detection results of the panoramic image. The detection results include target detection boxes and detection confidence scores.

[0043] Step S3: Perform semantic understanding on each object detection box to obtain the object category and category confidence corresponding to each object detection box.

[0044] Step S4: Backproject the target detection box of each perspective sub-image to the coordinate system of the panoramic image and filter it to obtain a first pseudo-label set; and obtain a second pseudo-label set based on the detection results and semantic understanding results of the panoramic image.

[0045] Step S5: Perform a fusion process on the first pseudo-label set and the second pseudo-label set, and obtain the pseudo-label data of the panoramic image based on the fusion result.

[0046] In some embodiments, step S3 obtains the target category and category confidence level corresponding to each target detection box through the following steps:

[0047] For each object detection box, a corresponding image region is cropped from the perspective sub-image based on the object detection box. The image region is then input into a pre-trained visual language generation model to generate a text description of the image region. Natural language processing is performed on the text description to extract at least one noun phrase as a candidate category. The image region is then input into the image encoder of the pre-trained visual language model to obtain an image feature vector. Each candidate category is then input into the text encoder of the visual language model to obtain a corresponding text feature vector. The similarity between the image feature vector and each text feature vector is calculated. The candidate category with the highest similarity is selected as the target category of the object detection box, and the similarity is used as the category confidence score.

[0048] In some embodiments, step S4 obtains the first set of pseudo-tags through the following steps:

[0049] For each target detection box in the perspective sub-image, uniform sampling is performed on its four sides to obtain multiple boundary sampling points; each boundary sampling point is mapped to the coordinate system of the panoramic image to obtain the corresponding projection point set; when adjacent projection points satisfy the condition of crossing the longitude boundary line, the polyline segment formed by the adjacent projection points is segmented; for the set of polyline segments formed by adjacent projection points in the projection point set, its minimum bounding rectangle is calculated as the back-projection detection box of the target detection box in the panoramic image; the back-projection detection boxes are filtered according to their position information and included in the first pseudo-label set.

[0050] Wherein, the adjacent projection points satisfy the condition of crossing the longitude boundary line, including:

[0051] Obtain the horizontal coordinates of two adjacent projection points in the panoramic image coordinate system. When the absolute value of the difference between the horizontal coordinates of two adjacent projection points is greater than half the width of the panoramic image, it is determined that the adjacent projection points meet the condition of crossing the longitude boundary.

[0052] In some embodiments, step S5 includes fusing the first pseudo-tag set and the second pseudo-tag set through the following steps:

[0053] Obtain matching target detection bounding box pairs from the first pseudo-label set and the second pseudo-label set; determine fusion weights based on the detection confidence of the target detection bounding box pairs and their positional features in the panoramic image; perform fusion processing on the target detection bounding box pairs based on the fusion weights to obtain fused target detection bounding boxes.

[0054] The fused target detection box is represented by the following formula:

[0055] (1)

[0056] in, For the i-th target detection box in the first set of pseudo-labels, Let j be the j-th target detection box in the second pseudo-label set. Let i and j be a pair of matching target detection boxes. Let i be the weight factor corresponding to the i-th target detection box. Let be the detection confidence score of the i-th target detection box. Let j be the weight factor corresponding to the j-th target detection box. Let be the detection confidence score of the j-th target detection box. The distortion coefficient is... The target detection box is obtained by fusing the i-th target detection box and the j-th target detection box.

[0057] The distortion coefficient This can be expressed by the following formula:

[0058] (2)

[0059] in, Let be the spherical latitude corresponding to the j-th target detection box. This is a preset lower bound constant for the weights, used to avoid the weights being too low. Let j be the y-coordinates of the center points of the target detection boxes. The height of the panoramic image is given.

[0060] To characterize the degree of geometric distortion of panoramic images in different latitude regions, this embodiment introduces a method based on... The distortion adjustment mechanism automatically reduces the contribution weight of ERP detection results in polar regions, thereby mitigating positioning errors caused by panoramic projection distortion.

[0061] Next, the panoramic image pseudo-tag generation process of the embodiments of this application will be described in detail.

[0062] First, collect panoramic image datasets in ERP format.

[0063] This embodiment can collect publicly available panoramic datasets corresponding to panoramic object detection tasks from multiple data sources, including but not limited to ERA, WHU, 360-indoor, and PANDORA datasets. The collected panoramic datasets are divided into training and validation sets in a 7:3 ratio and then merged.

[0064] Next, a geometric transformation is performed on the panoramic image.

[0065] The purpose of this step is to map the panoramic image in ERP format into perspective sub-images of a cube in six perspective directions.

[0066] Assuming the panoramic image in ERP format has a size of The output resolution for each face of the cube is For each cube face in the normalized two-dimensional planar coordinate system, for any pixel index (i,j) within the face, its normalized coordinates are defined as follows:

[0067] (3)

[0068] In formula (3), .

[0069] Based on the spatial orientation of different cube faces, the planar coordinates (u,v) are mapped to three-dimensional spatial direction vectors (x,y,z). Taking the current face as the front face as an example, then... If the front view is the back view, then If the current view is the right view, then... If the current view is the left view, then... If the current plane is the top plane, then... If the current view is the bottom view, then... .

[0070] To ensure that the direction vector lies on the unit sphere, the above vector is normalized:

[0071] (4)

[0072] Convert the unit direction vector into latitude and longitude parameters in a spherical coordinate system, where longitude... with latitude Calculated separately as

[0073] (5)

[0074] According to the definition of the ERP projection model, the spherical latitude and longitude can be mapped back to the pixel coordinates of the original ERP image:

[0075] (6)

[0076] In this embodiment, a periodic boundary condition is applied to the horizontal direction of the original ERP image pixel coordinate system, that is, for Modular operations are performed to ensure longitude continuity, and bilinear interpolation is used in the original panoramic image. upper coordinates Resampling is performed to obtain the pixel values ​​on the corresponding faces of the cube, which can be formally expressed as:

[0077] (7)

[0078] In formula (7), Let be the pixel value at (i,j) in the original panoramic image. This is the operator for the bilinear interpolation algorithm.

[0079] Thus, through the steps described above in this embodiment, the same inverse spherical mapping and interpolation sampling process can be performed on each cube face, ultimately obtaining six images of size [missing information]. The perspective projection image corresponds to six directions: front, back, left, right, top, and bottom. This embodiment constructs a spherical back-projection relationship using unit direction vectors, achieving a precise geometric transformation from equidistant cylindrical projection to a cubic perspective image, while ensuring continuity at the projection boundaries and the quality of image reconstruction.

[0080] Then, target detection is performed.

[0081] A pre-trained open-vocabulary object detection model was used to detect objects in the six perspective sub-images and the original panoramic image. The open-vocabulary object detection model in this embodiment is pre-trained on large-scale image-text pair data and possesses class-independent object detection capabilities.

[0082] For example, six perspective sub-images are sequentially input into the open-vocabulary object detection model, which outputs the detected bounding boxes and detection confidence scores for each perspective sub-image. Similarly, the original panoramic image is input into the open-vocabulary object detection model, which outputs the detected bounding boxes and detection confidence scores for the objects in the original panoramic image.

[0083] Since each perspective subgraph is a planar image that conforms to the laws of perspective, pre-trained open vocabulary object detection models can achieve better performance on such images.

[0084] Next, category detection is performed.

[0085] After obtaining six perspective sub-images and the target detection boxes of the original panoramic image, semantic descriptions are generated and visual semantic consistency is verified for each target detection box to determine the target category and its category confidence.

[0086] Specifically, for each input image, the corresponding set of object detection boxes is read. Each object detection box is represented as Based on the coordinates of the target detection box, the corresponding image region is cropped from the input image. The cropped image area Together with a pre-set prompt, such as "Name the main thing within the area.", the prompt is input into a pre-trained visual language generation model, such as the BLIP2-Flan-T5-XL model, to generate natural language description text for the corresponding area. This process can be formally represented as:

[0087] (8)

[0088] in, This represents a visual language generation model, where P is a pre-defined prompt.

[0089] For generated text Natural language processing is performed, and part-of-speech tagging and noun phrase extraction methods are used to extract a set of noun phrases. This serves as the set of candidate words for the category of the object detection box. If no valid noun phrases are extracted, the candidate region is discarded.

[0090] In this embodiment, to verify the consistency between visual and semantic data, the image region is... The image encoder input to CLIP yields the image feature vector. At the same time, each phrase in the candidate noun phrase set Inputting the CLIP text encoder yields the corresponding text feature vector. The image feature vectors and text feature vectors are both subjected to L2 normalization.

[0091] (9)

[0092] In formula (9), This represents the normalized image feature vector. This represents the normalized text feature vector.

[0093] Among them, the target detection box Cosine similarity between image feature vectors and text feature vectors:

[0094] (10)

[0095] For the target detection box Select the noun phrases with the highest similarity. It uses the target category as its target category and the maximum similarity as its category confidence score. , , .

[0096] After completing the similarity calculation, set a similarity threshold. Only when When the target detection box and its corresponding target category are retained, the target detection box and its corresponding target category are retained; if If the visual meaning does not match the semantic meaning, it will be rejected.

[0097] In this way, we can obtain a second set of pseudo-labels containing the coordinates of the target detection box, the detection confidence, the target category, and the category confidence, as well as a set of labels for six perspective sub-graphs.

[0098] Then, the first set of pseudo-labels is constructed.

[0099] After obtaining the target detection boxes, class labels, detection confidence, and class confidence of the six perspective sub-images, the target detection boxes on each perspective sub-image are reverse-mapped to the coordinate system of the original ERP panoramic image to generate the first set of pseudo-labels.

[0100] For example, suppose the bounding box for a target on a certain cube face is... Its coordinates are defined in a dimension of Within the plane of the cube face. In this embodiment, the target detection box may optionally be filtered by area if its planar area satisfies:

[0101] (11)

[0102] in, For example, a proportional threshold. Of course, it can also be set to other values.

[0103] If the target detection box The area is greater than If the target detection box is too large, it is considered to be a false detection area and will be removed.

[0104] For target detection boxes filtered by area, this embodiment does not directly project the four corner points, but instead performs uniform sampling along its four edges. Assuming the number of sampling points on each edge is N, then the boundary parameters... Discrete sampling is performed to obtain the set of boundary points. For any sampling point pixel coordinate ( First, convert it to normalized coordinates of a cube face:

[0105] (12)

[0106] After obtaining the normalized coordinates of the cube face, (u,v) is mapped to a three-dimensional direction vector (x,y,z) based on the current cube face orientation, defined in the same way as the forward projection. For example, if the current face is "front", then... This embodiment normalizes the three-dimensional direction vector to obtain the following form:

[0107] (13)

[0108] The spherical coordinates corresponding to the above three-dimensional direction vectors are as follows:

[0109] (14)

[0110] Furthermore, the above spherical coordinates are mapped to ERP image pixel coordinates as follows:

[0111] (15)

[0112] In the above formula, W and H are the dimensions of the original panoramic image.

[0113] After mapping all boundary sampling points of the target detection box using the above formula, the set of projected points on the ERP plane is obtained. Considering the longitude jump problem of the left and right boundaries of the ERP, if adjacent projected points satisfy:

[0114] (16)

[0115] The polyline formed by adjacent projection points is then segmented to avoid incorrect bounding box calculations caused by crossing longitude boundaries. For each valid projection point set, the minimum bounding rectangle is calculated, and the coordinates of its four corner points are as follows:

[0116]

[0117]

[0118] In practical applications, if the back-projected bounding box of the target detection box is close to both the left and right boundaries in the ERP coordinate system, then the following condition is met:

[0119] (17)

[0120] in, For boundary-tolerant pixels.

[0121] If the back-projection detection box is close to both the left and right boundaries in the ERP coordinate system, it is considered to cross the left and right boundaries of the ERP system and is therefore removed.

[0122] For all valid back-projection detection boxes, the back-projection detection boxes, target categories, detection confidence, and category confidence under their ERP coordinates are summarized, and the results from the six perspective sub-images are merged into the same ERP image label to obtain the first pseudo-label set.

[0123] Thus, this embodiment, through boundary sampling projection and segmented calculation of the circumscribed rectangle, can more accurately approximate the shape of the target area after spherical geometric transformation, improve the positioning accuracy of the panoramic pseudo-label, and avoid the frame distortion problem caused by longitude jumps.

[0124] Next, the first set of pseudo-tags and the second set of pseudo-tags are merged.

[0125] In this embodiment, after obtaining the first pseudo-label set and the second pseudo-label set, the pseudo-labels generated by the two different paths are uniformly modeled and fused to construct a more complete, accurate and robust panoramic pseudo-label set.

[0126] Specifically, for any ERP image, assume that the first set of pseudo-labels generated by the six-sided projection path of the cube and back-projected onto the ERP coordinate system is as follows: The second set of pseudo-labels obtained by directly inputting the original panoramic image into the detection model is... Where b represents the target detection box in ERP coordinates, Indicates the confidence level of the detection. Indicates category confidence. Indicates the corresponding target category, This indicates the number of targets in the first set of pseudo-labels. This indicates the number of targets in the second set of pseudo-labels.

[0127] Perform pairwise matching on two sets in the ERP coordinate system, for any... and The Intersection over Union (IoU) ratio is calculated using the following formula:

[0128] (18)

[0129] In formula (18), This is a preset threshold.

[0130] when and When the IoU value is greater than a preset threshold, the two detection boxes are considered to correspond to the same target instance. For successfully matched detection box pairs, a confidence-based weighted fusion strategy is introduced to optimize their spatial location, resulting in a fused target detection box. The coordinates are calculated using the formula (1) mentioned above, and will not be repeated here in this embodiment.

[0131] In practical applications, to enhance adaptability to targets of different scales, this embodiment introduces a selective fusion strategy based on target scale.

[0132] Specifically, when fusing the target detection box pairs based on the fusion weights, if the area of ​​any target detection box in the target detection box pair is less than a preset small target area threshold, then the target detection box in the first pseudo-label set is used as the fused target detection box; if the target detection box in the target detection box pair belonging to the first pseudo-label set is located at the projection plane boundary, then the target detection box in the second pseudo-label set is used as the fused target detection box. For other types of target detection box pairs, the fused target detection box is calculated based on formula (1).

[0133] The area function of the target detection box is:

[0134] (19)

[0135] When satisfied At the same time, the detection results of the cube projection path are retained to improve the detection recall rate of small targets. Here... This is the threshold for the small target area.

[0136] It is worth noting that for target detection boxes that fail to match, they are directly retained in the fusion result to ensure the recall rate of candidate targets.

[0137] Thus, through the above matching, weighted fusion, and adaptive selection strategies, a fused pseudo-label set is obtained, which is expressed in the following form:

[0138]

[0139] in, To merge the target number in the pseudo-label set, , .

[0140] This embodiment fully utilizes the structural differences in error distribution between the cube projection path and the ERP direct detection path through the cross-path pseudo-label fusion method described above. By introducing a path-aware weight adjustment mechanism, it achieves synergistic optimization of target positioning accuracy, small target detection capability, and target integrity, thereby significantly improving the overall quality and robustness of panoramic pseudo-labels.

[0141] Finally, pseudo-label filtering is performed to obtain the final pseudo-label data for the panoramic image.

[0142] In practical applications, each sample in panoramic datasets such as ERA, WHU, 360-indoor, and PANDORA is usually equipped with labeled data. Therefore, it is necessary to deduplicatize the fused pseudo-label set and the original labeled data to obtain the final pseudo-label data that can be used for model training.

[0143] Specifically, the target detection boxes in the fused pseudo-label set are matched one by one with the bounding boxes in the original annotation data. For example, for each target detection box, the IoU value between it and the bounding box is calculated. When the IoU value between the target detection box and any other target detection box is greater than a preset threshold, it is determined that the target detection box and the bounding box have a duplicate or highly overlapping relationship. The target detection box is then removed from the fused pseudo-label set to avoid semantic conflicts or duplicate supervision issues between real annotations and pseudo-labels.

[0144] After completing the above deduplication process, the remaining fused pseudo-label set is the final panoramic pseudo-label dataset. The pseudo-label data can be used together with the real labeled data for subsequent training of the object detection model, thereby expanding the training sample size and improving the model's generalization ability to open categories.

[0145] Figure 3 This is a schematic diagram of an electronic device illustrated in this specification according to an exemplary embodiment. Please refer to... Figure 3 At the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, memory 308, a hardware acceleration device 310, and non-volatile memory 312, and may also include other hardware required for its functions. One or more embodiments of this application can be implemented in software, for example, the processor 302 reads the corresponding computer program from the non-volatile memory 312 into memory 308 and then runs it. Of course, in addition to software implementation, one or more embodiments of this application do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the above processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0146] Figure 4 This is a structural block diagram of a panoramic image pseudo-tag generation device based on six-view projection, as illustrated in an exemplary embodiment of this application. The panoramic image pseudo-tag generation device can be applied to, for example... Figure 4 The electronic device shown implements the technical solution of this application. The panoramic image pseudo-tag generation device includes: a projection transformation unit 410, a target detection unit 420, a category detection unit 430, a tag integration unit 440, and a pseudo-tag generation unit 450, wherein:

[0147] The projection transformation unit 410 is used to perform geometric transformation on the panoramic image in equidistant cylindrical projection ERP format to obtain perspective sub-images from different viewpoints.

[0148] The object detection unit 420 is used to perform object detection on each perspective sub-image and the panoramic image using a pre-trained open vocabulary detection model, and obtain the detection results of each perspective sub-image and the detection results of the panoramic image. The detection results include object detection boxes and detection confidence scores.

[0149] The category detection unit 430 is used to perform semantic understanding on each target detection box to obtain the target category and category confidence corresponding to each target detection box;

[0150] The label integration unit 440 is used to backproject the target detection box of each perspective sub-image to the coordinate system of the panoramic image and filter it to obtain a first pseudo-label set; and to obtain a second pseudo-label set based on the detection results and semantic understanding results of the panoramic image.

[0151] The pseudo-label generation unit 450 is used to perform fusion processing on the first pseudo-label set and the second pseudo-label set, and obtain pseudo-label data of the panoramic image based on the fusion result.

[0152] In some embodiments, the pseudo-label generation unit 450 is configured to acquire target detection box pairs that match each other in the first pseudo-label set and the second pseudo-label set; determine fusion weights based on the detection confidence of the target detection box pairs and their positional features in the panoramic image; and perform fusion processing on the target detection box pairs based on the fusion weights to obtain fused target detection boxes.

[0153] In some embodiments, the category detection unit 430 is configured to, for each target detection box, crop the corresponding image region from the perspective sub-image where the target detection box is located, input the image region into a pre-trained visual language generation model to generate a text description of the image region; perform natural language processing on the text description to extract at least one noun phrase as a candidate category; input the image region into the image encoder of the pre-trained visual language model to obtain an image feature vector, and input each of the candidate categories into the text encoder of the visual language model to obtain a corresponding text feature vector; calculate the similarity between the image feature vector and each of the text feature vectors, select the candidate category with the highest similarity as the target category of the target detection box, and use the similarity as the category confidence score.

[0154] In some embodiments, the label integration unit 440 is configured to uniformly sample each target detection box in the perspective sub-image along its four sides to obtain multiple boundary sampling points; map each boundary sampling point to the coordinate system of the panoramic image to obtain a corresponding set of projection points; when adjacent projection points satisfy the condition of crossing a longitude boundary line, segment the polyline segment formed by the adjacent projection points; calculate the minimum bounding rectangle of the set of polyline segments formed by adjacent projection points in the set of projection points, and use it as the back-projection detection box of the target detection box in the panoramic image; filter the back-projection detection boxes according to their position information, and include the filtered back-projection detection boxes into the first pseudo-label set.

[0155] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0156] Accordingly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0157] Accordingly, embodiments of this application also provide a computer program product configured to perform the methods described in any of the above embodiments.

[0158] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0159] In a typical configuration, a computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0160] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0161] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0162] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0163] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0164] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0165] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0166] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating pseudo-tags for panoramic images based on six-view projection, characterized in that, Includes the following steps: Step S1: Perform geometric transformation on the panoramic image in equidistant cylindrical projection ERP format to obtain perspective sub-images from different viewpoints. Step S2: Using a pre-trained open vocabulary detection model, target detection is performed on each perspective sub-image and the panoramic image respectively, to obtain the detection results of each perspective sub-image and the detection results of the panoramic image. The detection results include target detection boxes and detection confidence scores. Step S3: Perform semantic understanding on each target detection box to obtain the target category and category confidence corresponding to each target detection box; Step S4: Back-project the target detection box of each perspective sub-image to the coordinate system of the panoramic image and filter it to obtain the first pseudo-label set; Based on the detection results and semantic understanding results of the panoramic image, a second set of pseudo-labels is obtained; Step S5: Perform a fusion process on the first pseudo-label set and the second pseudo-label set, and obtain the pseudo-label data of the panoramic image based on the fusion result.

2. The method according to claim 1, characterized in that, Step S5 includes: Obtain the target detection box pairs that match each other in the first pseudo-label set and the second pseudo-label set; The fusion weights are determined based on the detection confidence of the target detection box pairs and their positional features in the panoramic image. The target detection boxes are fused based on the fusion weights to obtain fused target detection boxes.

3. The method according to claim 2, characterized in that, The fused target detection box is represented by the following formula: in, For the i-th target bounding box in the first set of pseudo-labels, Let j be the j-th target detection box in the second pseudo-label set. Let i and j be a pair of matching target detection boxes. Let i be the weight factor corresponding to the i-th target detection box. Let be the detection confidence score of the i-th target detection box. Let j be the weight factor corresponding to the j-th target detection box. Let be the detection confidence score of the j-th target detection box. The distortion coefficient is... The target detection box is obtained by fusing the i-th target detection box and the j-th target detection box.

4. The method according to claim 3, characterized in that, The distortion coefficient This can be expressed by the following formula: in, Let be the spherical latitude corresponding to the j-th target detection box. The lower bound constant of the preset weights, Let j be the y-coordinates of the center points of the target detection boxes. The height of the panoramic image is given.

5. The method according to claim 1, characterized in that, Step S3 includes: For each object detection box, the corresponding image region is cropped from the perspective sub-image based on the object detection box, and the image region is input into the pre-trained visual language generation model to generate a text description of the image region; Natural language processing is performed on the text description to extract at least one noun phrase as a candidate category; The image region is input into the image encoder of the pre-trained visual language model to obtain the image feature vector, and each candidate category is input into the text encoder of the visual language model to obtain the corresponding text feature vector. Calculate the similarity between the image feature vector and each of the text feature vectors, select the candidate category with the highest similarity as the target category of the target detection box, and use the similarity as the category confidence score.

6. The method according to claim 1, characterized in that, Step S4 includes: For each target detection box in the perspective sub-image, uniform sampling is performed on its four sides to obtain multiple boundary sampling points; Each of the boundary sampling points is mapped to the coordinate system of the panoramic image to obtain the corresponding set of projection points; When adjacent projection points meet the condition of crossing the longitude boundary line, the broken line segment formed by the adjacent projection points is segmented. For the set of polyline segments formed by adjacent projection points in the set of projection points, calculate its smallest bounding rectangle, which is used as the back-projection detection box of the target detection box in the panoramic image. Based on the position information of the back-projection detection boxes, the back-projection detection boxes that pass the screening are included in the first pseudo-label set.

7. The method according to claim 6, characterized in that, The adjacent projection points satisfy the condition of crossing the longitude boundary line, including: Obtain the horizontal coordinates of two adjacent projection points in the panoramic image coordinate system; When the absolute value of the difference between the horizontal coordinates of two adjacent projection points is greater than half the width of the panoramic image, the adjacent projection points are determined to meet the condition of crossing the longitude boundary.

8. A panoramic image pseudo-tag generation device based on six-view projection, characterized in that, include: The projection transformation unit is used to perform geometric transformations on panoramic images in equidistant cylindrical projection ERP format to obtain perspective sub-images from different viewpoints. The object detection unit is used to perform object detection on each perspective sub-image and the panoramic image using a pre-trained open vocabulary detection model, and to obtain the detection results of each perspective sub-image and the detection results of the panoramic image. The detection results include object detection boxes and detection confidence scores. The category detection unit is used to perform semantic understanding on each target detection box to obtain the target category and category confidence of each target detection box. The label integration unit is used to backproject the target detection box of each perspective sub-image to the coordinate system of the panoramic image and filter them to obtain the first pseudo-label set. Based on the detection results and semantic understanding results of the panoramic image, a second set of pseudo-labels is obtained; The pseudo-label generation unit is used to perform fusion processing on the first pseudo-label set and the second pseudo-label set, and obtain pseudo-label data of the panoramic image based on the fusion result.

9. An electronic device, characterized in that, include: processor; A computer-readable storage medium storing computer program instructions that, when executed by the processor, cause the processor to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is executed by a processor according to any one of claims 1 to 7.

Images