Panoramic image feature extraction method and device based on self-supervised learning

Abstract

The invention provides a panoramic image feature extraction method and device based on self-supervised learning, and the method comprises the steps: enabling a plane image to be tangent to a unit sphere, projecting all pixel points of the plane image to two designated positions of the unit sphere, and obtaining two equidistant rectangular projections of the plane image; cutting each equidistant rectangular projection to generate a pair of training views with panoramic distortion; inputting each training view into an encoder to obtain an output feature of each training view, and inputting the output features into a convolutional layer to generate an offset position matrix; sampling coordinates are obtained according to the offset position matrix, grid sampling is carried out on the output features, and resampling features with deformation perception are obtained; and measuring the consistency with the output features of another training view by the resampling features of each training view, and constructing a loss function training encoder to obtain a panoramic feature extractor.

Description

Technical Field

[0001] This invention relates to the fields of image processing and machine vision, and particularly to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for panoramic image feature extraction based on self-supervised learning. Background Technology

[0002] Panoramic images offer a 360-degree field of view and provide a wealth of information about a scene at once. The most common representation of panoramic images is generated through equidistant rectangular projection (ERP), which is also the standard format for panoramic image datasets. Due to their ultra-wide field of view, annotating panoramic image datasets is typically more time-consuming and expensive than annotating planar images. This leads to a lack of data for various vision tasks based on panoramic images. Currently, two types of methods apply self-supervised learning to alleviate the data scarcity problem in panoramic images: planar methods and panoramic panorama-based methods. Figure 1 As shown, the planar-based method Plane trains a more robust backbone network using self-supervised learning on a standard planar image dataset, and then transfers it to a model for panoramic images. Figure 1 As shown, the panorama-based method utilizes unlabeled panoramic images, enabling the encoder to learn high-quality representations by maximizing the mutual information between different viewpoints at the equator and poles. The planar method takes a planar image as input and generates two distortion-free views using random cropping. The panorama-based method takes a panoramic image (displayed on a sphere) as input and rotates the sphere to generate two isometric rectangular views from different viewpoints.

[0003] However, regardless of whether the input is a panoramic image or a planar image, the above methods do not take into account the inherent distortion problem of panoramic images, which hinders the backbone network from learning the distortion features of panoramic images. For example, the planar method takes an image without distortion as input, so it cannot learn distortion features; the panoramic method is problematic because panoramic distortion is latent, and maximizing the mutual information between the equator and the poles alone cannot reflect the distortion features. Summary of the Invention

[0004] The purpose of this invention is to address the problem that current self-supervised learning methods for panoramic images do not consider panoramic image distortion, thus hindering the improvement of model feature representation capabilities. A self-supervised method for panoramic images is proposed.

[0005] To address the shortcomings of existing technologies, such as Figure 6 As shown, this invention proposes a panoramic image feature extraction method based on self-supervised learning, which includes:

[0006] The projection step involves tangenting the planar image containing the target object in the picture to the unit sphere, and projecting all the pixels of the planar image onto the unit sphere at two specified positions to obtain two equidistant rectangular projections of the planar image.

[0007] The cropping step involves cropping each of the equidistant rectangular projections to generate a pair of training views with panoramic distortion.

[0008] In the encoding step, each training view is input into the encoder to obtain the output features of each training view. The output features are then input into a convolutional layer to generate an offset position matrix. The sampling coordinates are obtained based on the offset position matrix, and the output features are subjected to grid sampling to obtain resampled features with deformation awareness.

[0009] The training step involves constructing a loss function to train the encoder by measuring the consistency with the output features of another training view using the resampled features of each training view, thus obtaining the panoramic feature extractor.

[0010] The extraction step involves inputting the panoramic image containing the target object, from which features to be extracted, into the panoramic image feature extractor to obtain the panoramic features of the panoramic image.

[0011] The aforementioned panoramic image feature extraction method based on self-supervised learning includes the following extraction steps:

[0012] The panoramic features are input into the semantic segmentation model to complete the semantic segmentation task of the panoramic image, and the semantic segmentation result of the panoramic image is obtained. The semantic segmentation result indicates whether each pixel in the panoramic image belongs to the target object or the background.

[0013] The panoramic image feature extraction method based on self-supervised learning, wherein the cropping step includes:

[0014] Center point e of the projected tangent P (θ P ,φ P This can be converted to coordinates e on the projection E of the equidistant rectangle. P (x P ,y P ), at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula:

[0015] I(u,v)=E(i+u,j+v)

[0016] Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top left corner of the cropping area, respectively, and h and w are the height and width of the pre-scaled cropped image.

[0017] The encoding steps include:

[0018] Input the training view z i1 ,z i2 In this encoder, the output feature f:R C×H×W The output features are input into the convolutional layer c to generate the offset position o. ik =[o x ,o y Let ] represent the offset matrix, i represent the i-th image, k represent the k-th view, x represent the x-coordinate, and y represent the y-coordinate; where the position (x,y) satisfies x∈[0,W], y∈[0,H], and k=1 or 2; depending on the offset position, the pixel will be shifted from its original position, and this process can be represented by the following formula:

[0019]

[0020] Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn The new position on;

[0021] The training steps include:

[0022] Will I ′ ik (m,n) and the output feature f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, and are expressed as follows:

[0023]

[0024] Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function; the loss function is expressed as:

[0025]

[0026] Among them, L g and L d These are cosine similarity losses.

[0027] The panoramic image feature extraction method based on self-supervised learning includes the following encoding steps: inputting the output features into a convolutional layer to obtain a feature vector matrix, applying a linear shift to the feature vector matrix to generate the shift position matrix.

[0028] like Figure 7 As shown, this invention also proposes a panoramic image feature extraction device based on self-supervised learning, which includes:

[0029] The projection module tangents the planar image containing the target object in the screen to the unit sphere, and projects all the pixels of the planar image onto the unit sphere at two specified positions to obtain two equidistant rectangular projections of the planar image.

[0030] The cropping module crops each of the equidistant rectangular projections, generating a pair of training views with panoramic distortion;

[0031] The encoding module inputs each training view into the encoder to obtain the output features of each training view. The output features are then input into the convolutional layer to generate an offset position matrix. The sampling coordinates are obtained based on the offset position matrix, and the output features are subjected to grid sampling to obtain resampled features with deformation awareness.

[0032] The training module trains the encoder by constructing a loss function based on the resampled features of each training view, measuring the consistency with the output features of another training view, and obtaining the panoramic feature extractor.

[0033] The extraction module inputs the panoramic image from which features are to be extracted into the panoramic image feature extractor to obtain the panoramic features of the panoramic image.

[0034] The aforementioned panoramic image feature extraction device based on self-supervised learning, wherein the extraction module includes:

[0035] The panoramic features are input into the semantic segmentation model to complete the semantic segmentation task of the panoramic image and obtain the semantic segmentation result of the panoramic image.

[0036] The aforementioned panoramic image feature extraction device based on self-supervised learning, wherein the cropping module includes:

[0037] Center point e of the projected tangent P (θ P ,φ P This can be converted to coordinates e on the projection E of the equidistant rectangle. P (x P ,y P ), at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula:

[0038] I(u,v)=E(i+u,j+v)

[0039] Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top left corner of the cropping area, respectively, and h and w are the height and width of the pre-scaled cropped image.

[0040] The encoding module includes:

[0041] Input the training view z i1 ,z i2 In this encoder, the output feature f:R C×H×W The output features are input into the convolutional layer c to generate the offset position o. ik =[o x ,o y Let ] represent the offset matrix, i represent the i-th image, k represent the k-th view, x represent the x-coordinate, and y represent the y-coordinate; where the position (x,y) satisfies x∈[0,W], y∈[0,H], and k=1 or 2; depending on the offset position, the pixel will be shifted from its original position, and this process can be represented by the following formula:

[0042]

[0043] Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn The new position on;

[0044] This training module includes:

[0045] Will I ′ ik (m,n) and the output feature f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, and are expressed as follows:

[0046]

[0047] Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function; the loss function is expressed as:

[0048]

[0049] Among them, L g and L d These are cosine similarity losses;

[0050] The encoding module includes: inputting the output features into a convolutional layer to obtain a feature vector matrix, applying a linear shift to the feature vector matrix to generate the shift position matrix.

[0051] The present invention also proposes an electronic device, including the aforementioned panoramic image feature extraction device based on self-supervised learning. The electronic device may be connected to an information display device, which is used to display the semantic segmentation result with user-set display parameters, attributes, or through an artificial intelligence model.

[0052] The present invention also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the panoramic image feature extraction method based on self-supervised learning.

[0053] The present invention also proposes a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it implements the steps of the self-supervised learning-based panoramic image feature extraction method.

[0054] As can be seen from the above solutions, the advantages of the present invention are:

[0055] Overall, extensive experiments demonstrate that our proposed DASC-SPA method, after training and generating a backbone network and then fine-tuning it for downstream semantic segmentation visual tasks, achieves a significant performance improvement compared to baseline methods, showcasing the effectiveness and superiority of our approach. The technical results are as follows:

[0056] 1. This method significantly improves model performance on three panoramic image semantic segmentation datasets;

[0057] Comparative Experiments: We set up three groups of methods: 1) a supervised baseline method for reference, 2) a self-supervised method for planar images, and 3) a self-supervised method for panoramic images. We conducted a comprehensive comparison of our DASC-SPA method with the three existing methods on a panoramic dataset. Figure 3 As shown, our method outperforms state-of-the-art methods on all three datasets, demonstrating its effectiveness. For the SUN360E dataset, our method achieves improvements of 1.67% and 2.63% in mAcc and mIoU respectively compared to the second-best method in PPS. For the Stanford2D3DS and CVPG-Pano datasets, which have less object distortion, our method also achieves more satisfactory performance than other methods, highlighting the ability of our DASC-SPA method to handle the inherent distortion problem of panoramic images.

[0058] 2. In this method, the SPA strategy can improve data richness and enhance the model's robustness to distortion.

[0059] Figure 4 Experimental performance with and without the SPA strategy is demonstrated. Our SPA strategy, by generating data containing panoramic distortion and inputting it into the model for training, achieves significant improvements compared to the model without the SPA strategy. For the Mocov2, DenseCL, and Simsiam methods, it achieves mIoU gains of +1.24, +2.43, and +1.92, respectively, indicating that applying the SPA strategy is beneficial for general methods in panoramic images.

[0060] 3. In this method, the DASC framework can learn finer-grained distortion representations, thus improving robustness to distortion.

[0061] To verify the effectiveness of the DASC framework, we conducted experiments such as... Figure 5 As shown, it can be observed that adding the SPA strategy improves performance by 2.09 mAcc and 1.92 mIoU compared to the baseline scheme. When DASC and SPA are combined, the performance is significantly improved to 73.02 mAcc and 60.76 mIoU, demonstrating the superiority of the SPA strategy and the DASC framework. Attached Figure Description

[0062] Figure 1 A comparison chart of three self-supervised comparison methods;

[0063] Figure 2 This is a framework diagram of the DASC-SPA method of the present invention;

[0064] Figure 3 A comparison graph showing the different self-supervised methods on three panoramic datasets;

[0065] Figure 4 Histograms comparing different methods;

[0066] Figure 5 Figure for an ablation experiment to assess the effectiveness of different components;

[0067] Figure 6 This is a flowchart of the method of the present invention;

[0068] Figure 7 This is a block diagram of the device of the present invention;

[0069] Figure 8 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0070] Figure 9 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0071] Figure 10 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0072] Figure label:

[0073] A - First electronic device;

[0074] B-A panoramic image feature extraction device based on self-supervised learning;

[0075] C-Data acquisition equipment;

[0076] D-Information display device;

[0077] 1000 - Second electronic device;

[0078] Ⅰ-Computational Unit;

[0079] II-ROM;

[0080] III-RAM;

[0081] N-bus;

[0082] V-Interface;

[0083] VI - Input Unit;

[0084] VII - Output Unit;

[0085] VIII - Storage medium;

[0086] IX - Communication Unit. Detailed Implementation

[0087] It should be noted that, in this application, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0088] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0089] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0090] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0091] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0092] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0093] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0094] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0095] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0096] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0097] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0098] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0099] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0101] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] To introduce panoramic distortion and enable the backbone network to learn distortion information, this invention proposes a Spherical Projection Augmentation (SPA) strategy, such as... Figure 2As shown on the left, this includes spherical projection and center cropping operations. Spherical projection maps a planar image to different latitudes, generating panoramic distortions of varying degrees. Center cropping filters out redundant information and preserves object distortion by cropping the projected image based on the image center. By using SPA, views with different distortions can be generated from the same planar image while maintaining the original image size. We demonstrate that applying SPA to current self-supervised learning methods can improve the performance of downstream tasks. Since the SPA policy introduces panoramic distortion, pixels in different views are misaligned except for the center pixel. We propose a Deformation-aware Sampling Consistency (DASC) framework, introducing an additional dense pixel learning mechanism. We utilize two convolutional layers to predict the offset of each pixel in one view. Based on their position, each pixel is mesh-sampled using the predicted offset, aligning it with the corresponding pixel in another view. This approach aims to achieve pixel-level deformation-aware consistency, enabling the backbone network to learn finer distortion representations and improving robustness to distorted perspectives.

[0103] In summary, to achieve the above-mentioned technical effects, the present invention proposes the following key technical points:

[0104] Key point 1: This method proposes the SPA strategy, which introduces panoramic distortion, makes full use of existing planar datasets, solves the problem of panoramic data scarcity, and preserves the characteristics of distortion features.

[0105] Key point 2: This method proposes the DASC framework, which aims to predict pixel offsets and perform mesh sampling to achieve pixel-level deformation-aware consistency and make it robust to panoramic distortion.

[0106] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0107] The overall structure of this method is as follows: Figure 2 As shown, it includes the SPA strategy for data preprocessing and the DASC framework for network structure. Figure 2 The final framework obtained is the backbone network. Self-supervised contrastive learning methods mainly utilize their respective innovations to learn from unlabeled data and obtain a weighted backbone network. Figure 2The Encoder in this context is the backbone network. The backbone network can subsequently participate in many image processing tasks such as image classification, segmentation, and detection. Its role is to provide the model with robust feature extraction capabilities and to learn general representations that can be transferred to various tasks. Therefore, this application aims to obtain a backbone network that is robust to panoramic distortion and can enhance the feature extraction capabilities of panoramic images.

[0108] 1) Spherical projection enhancement strategy:

[0109] Using panoramic images as input presents significant challenges due to uncontrollable issues such as random variations in the number of objects and distortion. Therefore, this invention proposes a SPA (Special Spatial Perspective) strategy that generates controllable panoramic distortion from planar images for self-supervised learning of the distortion. Figure 2 As shown in the left part, the strategy proposed in this invention consists of two steps: spherical projection and center clipping. Spherical projection involves tangenting the planar image to a unit sphere S and rotating the sphere so that the projection point (the point of tangency) is aligned with the x-axis, which is the horizontal axis in a three-dimensional coordinate system constructed with the center of the unit sphere as the origin. Then, all points of the planar image are projected onto the surface of the unit sphere S. The unit sphere S is a sphere with a radius of 1, thereby reducing the variable factors of different sizes to the size of the image, that is, the field of view (Fov).

[0110] Spherical projection projects a planar image onto a sphere, transforming it into an equidistant rectangular projection ERP representation. The field of view (Fov) varies for different planar images, but it remains a planar image, generally not exceeding 60 degrees and certainly not 180 degrees, as distortion occurs beyond 60 degrees, similar to a fisheye image. We expect the planar image itself to be distortion-free, hence the Fov generally doesn't exceed 60 degrees. However, due to the narrow Fov, it contains a significant amount of irrelevant information. When using random cropping, the view may include background regions. Striving to make these views, containing redundant backgrounds, as similar as possible ultimately impacts model training. To address this, we further perform center cropping on the ERP representation to remove most of the background information, generating views with different distortions as input for self-supervised learning. Different degrees of distortion are obtained by projecting onto different points on the sphere. This can be achieved by... Figure 2 As shown, from when the projection point is aligned with the x-axis to when it moves along the xoz plane toward the z-axis, the shape of the projected image is different at any of these positions, which results in different distortions.

[0111] Specifically, the center point e of the projection tangent P (θ P ,φ P This can be converted to coordinates e on the ERP image E.P (x P ,y P We are at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula:

[0112] I(u,v)=E(i+u,j+v)

[0113] Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top-left corner of the cropped region, respectively, and h and w are the height and width of the pre-defined rescaled cropped image, respectively. Therefore, our center-cropping strategy can replace random cropping to generate paired views with different distortions, fully leveraging current large-scale planar image datasets for self-supervised learning.

[0114] 2) Deformation-aware sampling consistency framework:

[0115] Because the SPA strategy enables the model to adapt to target distortion in panoramic images, we propose the DASC framework. This framework takes paired views with different distortions as input, primarily aiming to improve the model's feature representation capabilities and robustness to distorted viewpoints by measuring the consistency between two different viewpoints after deformable space transformation. Figure 2 As shown on the right, α and β are parameters that can be adjusted manually. For the learned offset, we add a linear change, which is equivalent to manually adjusting the offset magnitude. Although the previous step generated an offset by projecting onto the unit sphere, this invention aims to artificially apply a certain influence, similar to adjusting parameters during model training, to allow the model to learn faster and perform better. DASC framework input pairwise views z i1 ,z i2 In the shared encoder, the output feature f:R C×H×W A shared encoder refers to a neural network where parameters are shared. These parameters are then input into two convolutional layers c to generate offset positions o. ik =[o x ,o yLet $\mathbf{x}$, where $o$ represents the offset matrix, $i$ represents the $i$-th image, $k$ represents the $k$-th view, $x$ represents the x-coordinate, and $y$ represents the y-coordinate. This offset position refers to the shift of each pixel in the feature map from its current position. For example, to find the position of the panda's nose in another panda view, the two positions may not be the same pixel coordinate. This coordinate difference is the offset we need to learn. The position (x, y) satisfies $x ∈ [0, W], $y ∈ [0, H]$, and $k = 1 or 2$, where 1 or 2 corresponds to feature Map1 or feature Map2, representing the offset between the two branches. Based on the predicted offset position, the pixel will shift from its original position. This process can be represented by the following formula.

[0116]

[0117] Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn A new position on the website.

[0118] Once the new deformable locations are calculated, these new coordinates will be used for mesh sampling. By doing so, the resampled feature map from one view can be used to measure consistency with another view. We will... ′ ik (m,n) and the encoded feature map f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, which is expressed as:

[0119]

[0120] Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function. By using this formula, we can obtain a deformability-aware resampled feature map to measure consistency with another view. We use cosine similarity as a consistency metric.

[0121] Therefore, the overall self-supervised training objective for panoramic images is represented as:

[0122]

[0123] Among them, L g and L dThese are cosine similarity losses. The former represents global optimization, while the latter represents deformation consistency quantification. Additionally, λ is the loss weight used to balance the two components, which is set to 1.0 by default in our experiments. The training target is the backbone network. i1 and z i2 yes Figure 2 The left side shows the paired views of the input. The purpose is to group distorted views from the same image closer together and distorted views from different images further apart, thus achieving a clustering effect and allowing the model to learn what distortions might cause shape changes to an object.

[0124] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0125] like Figure 7 As shown, this invention also proposes a panoramic image feature extraction device based on self-supervised learning, which includes:

[0126] The projection module tangents the planar image containing the target object in the screen to the unit sphere, and projects all the pixels of the planar image onto the unit sphere at two specified positions to obtain two equidistant rectangular projections of the planar image.

[0127] The cropping module crops each of the equidistant rectangular projections, generating a pair of training views with panoramic distortion;

[0128] The encoding module inputs each training view into the encoder to obtain the output features of each training view. The output features are then input into the convolutional layer to generate an offset position matrix. The sampling coordinates are obtained based on the offset position matrix, and the output features are subjected to grid sampling to obtain resampled features with deformation awareness.

[0129] The training module trains the encoder by constructing a loss function based on the resampled features of each training view, measuring the consistency with the output features of another training view, and obtaining the panoramic feature extractor.

[0130] The extraction module inputs the panoramic image from which features are to be extracted into the panoramic image feature extractor to obtain the panoramic features of the panoramic image.

[0131] The aforementioned panoramic image feature extraction device based on self-supervised learning, wherein the extraction module includes:

[0132] The panoramic features are input into the semantic segmentation model to complete the semantic segmentation task of the panoramic image and obtain the semantic segmentation result of the panoramic image.

[0133] The aforementioned panoramic image feature extraction device based on self-supervised learning, wherein the cropping module includes:

[0134] Center point e of the projected tangent P (θ P ,φ P This can be converted to coordinates e on the projection E of the equidistant rectangle. P (x P ,y P ), at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula:

[0135] I(u,v)=E(i+u,j+v)

[0136] Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top left corner of the cropping area, respectively, and h and w are the height and width of the pre-scaled cropped image.

[0137] The encoding module includes:

[0138] Input the training view z i1 ,z i2 In this encoder, the output feature f:R C×H×W The output features are input into the convolutional layer c to generate the offset position o. ik =[o x ,o y Let ] represent the offset matrix, i represent the i-th image, k represent the k-th view, x represent the x-coordinate, and y represent the y-coordinate; where the position (x,y) satisfies x∈[0,W], y∈[0,H], and k=1 or 2; depending on the offset position, the pixel will be shifted from its original position, and this process can be represented by the following formula:

[0139]

[0140] Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn The new position on;

[0141] This training module includes:

[0142] Will I ′ ik(m,n) and the output feature f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, and are expressed as follows:

[0143]

[0144] Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function; the loss function is expressed as:

[0145]

[0146] Among them, L g and L d These are cosine similarity losses;

[0147] The encoding module includes: inputting the output features into a convolutional layer to obtain a feature vector matrix, applying a linear shift to the feature vector matrix to generate the shift position matrix.

[0148] like Figure 8 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, which includes the aforementioned panoramic image feature extraction device B based on self-supervised learning.

[0149] like Figure 9 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire planar images or panoramic images, such as acquiring planar images with target objects during training and acquiring panoramic images taken by a panoramic camera during testing. The information display device D is used to display the training loss function results or semantic segmentation results obtained by the present invention.

[0150] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0151] The present invention also provides a computer program product, which includes a computer program that can be stored on a readable storage medium. When the computer program is executed by a processor, the computer is able to execute the panoramic image feature extraction method based on self-supervised learning provided by the above methods.

[0152] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that performs the aforementioned panoramic image feature extraction method based on self-supervised learning. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0153] Figure 10 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0154] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0155] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0156] The computing unit I can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S5. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0157] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A panoramic image feature extraction method based on self-supervised learning, characterized in that, include: The projection step involves tangenting the planar image to the unit sphere and projecting all pixels of the planar image onto the unit sphere at two specified positions to obtain two equidistant rectangular projections of the planar image. The cropping step involves cropping each of the equidistant rectangular projections to generate a pair of training views with panoramic distortion. In the encoding step, each training view is input into the encoder to obtain the output features of each training view. The output features are then input into a convolutional layer to generate an offset position matrix. The sampling coordinates are obtained based on the offset position matrix, and the output features are subjected to grid sampling to obtain resampled features with deformation awareness. The training step involves constructing a loss function to train the encoder by measuring the consistency with the output features of another training view using the resampled features of each training view, thus obtaining the panoramic feature extractor. The extraction step involves inputting the panoramic image from which features are to be extracted into the panoramic image feature extractor to obtain the panoramic features of the panoramic image.

2. The panoramic image feature extraction method based on self-supervised learning as described in claim 1, characterized in that, The extraction steps include: The panoramic features are input into the semantic segmentation model to complete the semantic segmentation task of the panoramic image and obtain the semantic segmentation result of the panoramic image.

3. The panoramic image feature extraction method based on self-supervised learning as described in claim 1, characterized in that, The cutting process includes: Center point e of the projected tangent P (θ P ,φ P This can be converted to coordinates e on the projection E of the equidistant rectangle. P (x P ,y P ), at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula: I(u,v)=E(i+u,j+v) Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top left corner of the cropping area, respectively, and h and w are the height and width of the pre-scaled cropped image. The encoding steps include: Input the training view z i1 ,z i2 In this encoder, the output feature f:R C×H×W The output features are input into the convolutional layer c to generate the offset position o. ik =[o x ,o y Let ] represent the offset matrix, i represent the i-th image, k represent the k-th view, x represent the x-coordinate, and y represent the y-coordinate; where the position (x,y) satisfies x∈[0,W], y∈[0,H], and k=1 or 2; depending on the offset position, the pixel will be shifted from its original position, and this process can be represented by the following formula: Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn The new position on; The training steps include: Will I' ik (m,n) and the output feature f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, and are expressed as follows: Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function; the loss function is expressed as: Among them, L g and L d These are cosine similarity losses.

4. The panoramic image feature extraction method based on self-supervised learning as described in claim 1, characterized in that, The encoding step includes: inputting the output features into a convolutional layer to obtain a feature vector matrix, applying a linear shift to the feature vector matrix to generate the shift position matrix.

5. A panoramic image feature extraction device based on self-supervised learning, characterized in that, include: The projection module tangents the planar image to the unit sphere and projects all pixels of the planar image onto the unit sphere at two specified positions, resulting in two equidistant rectangular projections of the planar image. The cropping module crops each of the equidistant rectangular projections, generating a pair of training views with panoramic distortion; The encoding module inputs each training view into the encoder to obtain the output features of each training view. The output features are then input into the convolutional layer to generate an offset position matrix. The sampling coordinates are obtained based on the offset position matrix, and the output features are subjected to grid sampling to obtain resampled features with deformation awareness. The training module trains the encoder by constructing a loss function based on the resampled features of each training view, measuring the consistency with the output features of another training view, and obtaining the panoramic feature extractor. The extraction module inputs the panoramic image from which features are to be extracted into the panoramic image feature extractor to obtain the panoramic features of the panoramic image.

6. The panoramic image feature extraction device based on self-supervised learning as described in claim 5, characterized in that, The extraction module includes: The panoramic features are input into the semantic segmentation model to complete the semantic segmentation task of the panoramic image and obtain the semantic segmentation result of the panoramic image.

7. The panoramic image feature extraction device based on self-supervised learning as described in claim 5, characterized in that, The trimming module includes: Center point e of the projected tangent P (θ P ,φ P This can be converted to coordinates e on the projection E of the equidistant rectangle. P (x P ,y P ), at point e P The surrounding image is cropped to obtain a rescaled cropped image I(u,v), which can be expressed by the following formula: I(u,v)=E(i+u,j+v) Where 0≤u <w,0≤v<h, i and j are the horizontal and vertical coordinates of the top left corner of the cropping area, respectively, and h and w are the height and width of the pre-scaled cropped image. The encoding module includes: Input the training view z i1 ,z i2 In this encoder, the output feature f:R C×H×W The output features are input into the convolutional layer c to generate the offset position o. ik =[o x ,o y Let ] represent the offset matrix, i represent the i-th image, k represent the k-th view, x represent the x-coordinate, and y represent the y-coordinate; where the position (x,y) satisfies x∈[0,W], y∈[0,H], and k=1 or 2; depending on the offset position, the pixel will be shifted from its original position, and this process can be represented by the following formula: Among them, I′ ik (m,n) represents the offset after transformation, starting from the (m,n) coordinates in the k-th view within the i-th image. Apply in the same position I mn The new position on; This training module includes: Will I' ik (m,n) and the output feature f are used as inputs to the grid sampling G based on differentiable bilinear interpolation, and are expressed as follows: Where p ik (x,y) represents the pixel value of the resampled feature map at position (x,y). This indicates that new horizontal and vertical coordinates are generated after deformation, represented by the symbol... This represents the floor function; the loss function is expressed as: Among them, L g and L d These are cosine similarity losses; The encoding module includes: inputting the output features into a convolutional layer to obtain a feature vector matrix, applying a linear shift to the feature vector matrix to generate the shift position matrix.

8. An electronic device, characterized in that, The device includes a panoramic image feature extraction apparatus based on self-supervised learning as described in claims 5-7. The electronic device may be connected to an information display device, which is used to display the semantic segmentation result with user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the panoramic image feature extraction method based on self-supervised learning as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the panoramic image feature extraction method based on self-supervised learning as described in any one of claims 1-4.

Images