Transformer-based panoramic image feature representation learning method

Abstract

The present invention provides a transformer-based panoramic image feature representation learning method, comprising: dividing a first panoramic image into a number of squares, extracting features from each square using a multi-layer deep convolutional neural network to obtain a first eigenvector for each square; dividing the squares into a number of first windows, performing a self-attention operation on all squares within each first window, and obtaining a second eigenvector for each square; subjecting all squares in the entire first panoramic image to a sliding transformation and a self-attention operation to obtain a third eigenvector for each square; rotating the copied first panoramic image by 90° to obtain a second panoramic image, performing a cross-attention operation, and obtaining a fourth eigenvector for each square; and, based on the obtained eigenvectors, merging the features of every four adjacent squares as a group to obtain processed features of the first panoramic image. This invention overcomes the spatial distortion and boundary discontinuity problems of panoramic images.

Description

Technical Field

[0001] The present invention relates to the technical field of panoramic image feature extraction, and in particular to a transformer-based panoramic image feature representation learning method. Background Art

[0002] In recent years, with the rapid development of technologies such as VR and autonomous driving, the application of panoramic images has become increasingly widespread. Panoramic image-related tasks continue to attract scholarly attention, including semantic segmentation, object detection, image classification, and indoor layout prediction. In the era of deep learning, these tasks are generally accomplished based on the depth features of panoramic images. Therefore, how to extract the depth features of panoramic images has become a critical issue. Current 360° panoramic images are generally represented using an equirectangular projection (ERP), which maps the longitude and latitude of spherical coordinates to the horizontal and vertical coordinates of a two-dimensional plane. However, this ERP mapping method poses significant challenges to panoramic image representation learning technology.

[0003] Although CNNs are widely used for feature extraction in two-dimensional images, traditional CNNs are unable to effectively extract features from panoramic images. This is because ERP introduces two problems: (1) boundary discontinuity. The South Pole or North Pole, which is originally a point, is mapped to a line, and the originally adjacent left and right boundaries are also mapped to the two ends of the panoramic image; (2) spatial distortion. Panoramic projection can cause distortion of objects, especially objects at high latitudes. Solving these two problems of panoramic images has become a hot topic in the study of panoramic image representation learning.

[0004] In order to extract the features of panoramic images, a common method is to adapt the convolution operation to the spherical space. When SphereNet performs convolution near a certain point, it creates a tangent plane at that point and projects the nearby pixels onto the plane for convolution; Kernel Transformer Network (KTN) uses a variety of different convolutions to adapt to distortions at different latitudes; S2CNN uses a Fourier transform filter to make the convolution satisfy the rotation consistency of the panoramic image; SphereNet samples uniformly on the sphere so that these points support planar convolution operations; SpherePHD uses polyhedrons to approximate the sphere, thereby converting spherical convolution into planar convolution; DeepSphere uses spherical convolution to make the model satisfy the rotation consistency of the panoramic image. These methods either have low computational efficiency due to the use of a large amount of sampling and projection, for example, the computation time of SphereNet is almost ten times that of a planar CNN; or, like SpherePHD, they destroy the original properties of the sphere.

[0005] Some ViT-based methods (such as PanoFormer and Sphrical Transformer) project the panoramic image back onto a sphere, then divide the sphere into different blocks, similar to the hexagonal blocks on the surface of a soccer ball, and then perform attention operations on these blocks. However, these methods are not compatible with two-dimensional images and cannot be combined with a large number of training methods based on two-dimensional images. They also do not fully utilize common pre-trained models to further improve model performance. Summary of the Invention

[0006] The purpose of this invention is to provide a transformer-based panoramic image feature representation learning method, which effectively overcomes the spatial distortion and boundary discontinuity problems brought about by existing panoramic image feature extraction methods, and makes up for the shortcomings of traditional CNN or ViT in panoramic image feature embedding.

[0007] To achieve the above objectives, the present invention provides a Transformer-based panoramic image feature representation learning method, comprising:

[0008] S1. Divide the first panoramic image into several squares, extract features of each square using a multi-layer deep convolutional neural network, and assign a Cartesian coordinate-based position code to each square feature to obtain a first feature vector of each square;

[0009] S2. Divide the first panoramic image into a plurality of first windows based on the squares, perform a local self-attention operation on the first eigenvectors of all squares in each first window, and obtain a second eigenvector of each square;

[0010] S3, subjecting all squares of the entire first panoramic image to a sliding transformation that supports the continuity of the boundaries of the first panoramic image to obtain a new square arrangement, repeating S2, and obtaining a third eigenvector of each square based on the second eigenvector of each square;

[0011] S4. Copy the first panoramic image once, rotate the copied first panoramic image by 90° in pitch to obtain a second panoramic image, transform the first window in the first panoramic image into the second window in the second panoramic image, perform a cross-attention operation on the second window in the second panoramic image and the first window in the first panoramic image, and obtain fourth eigenvectors corresponding to the respective squares in the original first panoramic image based on the third eigenvectors of the respective squares;

[0012] S5. Based on the fourth eigenvector of each square, the features of every four adjacent squares are combined as a group to obtain processed features of the first panoramic image.

[0013] Optionally, in S1, for each square, its center is represented by longitude u∈[-π,π] and latitude v∈[-0.5π,0.5π], and the Cartesian coordinates centered on the sphere center are obtained as follows:

[0014] x=sin(u)cos(v)

[0015] y=cos(u)cos(v)

[0016] z=-sin(v)

[0017] The five scalars x, y, z, u, and v are combined into a first vector, which is then encoded through a fully connected neural network layer to obtain a second vector with the same feature dimension. The first and second vectors are added together to obtain the first feature vector of each grid.

[0018] Optionally, after the first panoramic image is divided into several first windows based on the grid, the squares in the first window will be flattened into a row, thereby converting the first eigenvector of the grid into a two-dimensional matrix corresponding to the first window, and the two-dimensional matrix is ​​expressed as the number of squares * feature dimension.

[0019] Optionally, in the self-attention operation of S2, each pair of query Q i and key value k j After the dot product operation, a relative position offset based on the great circle distance will be added, where Q i represents the i-th row of the two-dimensional matrix, k j represents the j-th column of the two-dimensional matrix.

[0020] Optionally, the self-attention operation is set to a function attention, and the attention function is expressed as follows:

[0021]

[0022]

[0023] Where B ij represents the positional relationship between the i-th square and the j-th square, α ij represents the offset of the first panoramic image, β ij Indicates the offset of the plane image, u i and u i Represents the longitude and latitude of the i-th square, u j and v j They represent the longitude and latitude of the j-th square respectively, and Q, K, and V represent the two-dimensional matrices of the same first window.

[0024] Optionally, the step of subjecting all squares of the entire first panoramic image to a sliding transformation that supports boundary continuity of the first panoramic image specifically includes:

[0025] S31, circularly sliding the first panoramic image as a whole to the left by a distance of half the window size;

[0026] S32: vertically cut the first panoramic image in half, and rotate the right half 180° counterclockwise around the upper right corner of the left half;

[0027] S33: Circularly slide the first panoramic image as a whole downward by a distance of half the window size.

[0028] Optionally, in S3, after obtaining the third eigenvector of each square, the squares will be restored to their original order.

[0029] Optionally, in S4, for any point P(u, v) in the first panoramic image, assuming that after the North Pole of the first panoramic image is rotated from P0(0, -0.5π) to the target position P1(u1, v1), P(u, v) will be rotated to P′(u′, v′), and the pitch angle rotation operation is expressed by the following formula:

[0030] Sph(P(u a ,v a )):[sin(u a )cos(v a ),cos(u a )cos(v a )],-sin(v a )

[0031]

[0032]

[0033]

[0034] Among them, ":" means "definition", Represents the vector product, the Angle function gives and The positive direction of the angle is determined by Given.

[0035] Optionally, after S5, the transformer-based panoramic image feature representation learning method further includes:

[0036] S6. Repeat S2-S5 several times to obtain the final features of the first panoramic image.

[0037] Optionally, after S6, the transformer-based panoramic image feature representation learning method further includes:

[0038] S7. Use the final features obtained for downstream tasks.

[0039] In the transformer-based panoramic image feature representation learning method provided by the present invention, the problem of boundary discontinuity in panoramic images is overcome by adopting a sliding transformation that supports the continuity of panoramic image boundaries, and the problem of spatial distortion in panoramic images is overcome by position encoding based on Cartesian coordinates, relative position offset based on great circle distance, and an important pitch attention mechanism, thereby extracting high-quality panoramic image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Those skilled in the art will appreciate that the accompanying drawings are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

[0041] Figure 1 A diagram showing the steps of a transformer-based panoramic image feature representation learning method according to one embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the drawings are in a very simplified form and use non-precise proportions, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention. In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, please refer to the accompanying drawings. It should be noted that the structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification for people familiar with this technology to understand and read, and are not used to limit the conditions for the implementation of the present invention. Any modification of the structure, change in the proportional relationship or adjustment of the size, under the condition that the effect produced by the present invention and the purpose that can be achieved are the same or similar, should still fall within the scope of the technical content disclosed by the present invention.

[0043] As used in the present invention, the singular forms "a", "an" and "the" include plural objects, unless the content clearly indicates otherwise. As used in the present invention, the term "or" is generally used in a sense including "and / or", unless the content clearly indicates otherwise. As used in the present invention, the term "several" is generally used in a sense including "at least one", unless the content clearly indicates otherwise. As used in the present invention, the term "at least two" is generally used in a sense including "two or more", unless the content clearly indicates otherwise. In addition, the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first", "second" and "third" may explicitly or implicitly include one or at least two of the features.

[0044] In the description of the present invention, unless otherwise expressly specified or limited, the terms "mounted," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connection, detachable connection, or integration; mechanical connection or electrical connection; direct connection or indirect connection through an intermediate medium; and internal communication between two components or interaction between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention in specific circumstances.

[0045] Please refer to Figure 1 , Figure 1 This is a diagram of the steps of a transformer-based panoramic image feature representation learning method provided by one embodiment of the present invention. This embodiment of the present invention provides a transformer-based panoramic image feature representation learning method, including:

[0046] S1. Divide the first panoramic image into several squares, extract features of each square using a multi-layer deep convolutional neural network, and assign a Cartesian coordinate-based position code to each square feature to obtain a first feature vector of each square;

[0047] S2. Divide the first panoramic image into a plurality of first windows based on the squares, perform a local self-attention operation on the first eigenvectors of all squares in each first window, and obtain a second eigenvector of each square;

[0048] S3, subjecting all squares of the entire first panoramic image to a sliding transformation that supports the continuity of the boundaries of the first panoramic image to obtain a new square arrangement, repeating S, and obtaining a third eigenvector of each square based on the second eigenvector of each square;

[0049] S4. Copy the first panoramic image once, rotate the copied first panoramic image by 90° in pitch to obtain a second panoramic image, transform the first window in the first panoramic image into the second window in the second panoramic image, perform a cross-attention operation on the second window in the second panoramic image and the first window in the first panoramic image, and obtain fourth eigenvectors corresponding to the respective squares in the original first panoramic image based on the third eigenvectors of the respective squares;

[0050] S5. Based on the fourth eigenvector of each square, the features of every four adjacent squares are combined as a group to obtain processed features of the first panoramic image.

[0051] First, for a given first panoramic image (i.e., the initial image), S1 is executed to divide the first panoramic image into several squares. The features of each square are extracted using a multi-layer deep convolutional neural network. These square features are then encoded with a Cartesian coordinate position code to obtain the first feature vector of each square. In S1, each square has a longitude and latitude coordinate at its center, which can be expressed as longitude u∈[-π,π] and latitude v∈[-0.5π,0.5π]. The Cartesian coordinates centered on the sphere center are as follows:

[0052] x=sin(u)cos(v)

[0053] y=cos(u)cos(v)

[0054] z=-sin(v)

[0055] The five scalars x, y, z, u, and v are combined into a first vector, which is then encoded through a fully connected neural network layer to obtain a second vector with the same feature dimension. The first and second vectors are added together to obtain the first feature vector of each grid.

[0056] Then, S2 is executed to divide the first panoramic image into a plurality of first windows based on the squares, and a local self-attention operation is performed on the first eigenvectors of all squares within each first window to obtain a second eigenvector of each square. In this embodiment, the squares of the first panoramic image are divided into first windows of fixed size, and the squares within these first windows are flattened into a row, thereby converting the first eigenvectors of the squares into a two-dimensional matrix corresponding to the first window. The two-dimensional matrix is ​​represented as the number of squares * feature dimension, and the self-attention operation can then be performed on the two-dimensional matrix.

[0057] In the self-attention operation of S2, each pair of query Q i and key value k jAfter the dot product operation, a relative position offset based on the great circle distance will be added, where Q i represents the i-th row of the two-dimensional matrix, k j The great circle distance query is the great circle distance between the two squares corresponding to the key value.

[0058] Specifically, the self-attention operation is set as the function attention, and the attention function is expressed as follows:

[0059]

[0060]

[0061] Where B ij represents the positional relationship between the i-th square and the j-th square, α ij represents the offset of the first panoramic image, β ij Indicates the offset of the plane image, u i and v i Represents the longitude and latitude of the i-th square, u j and v j They represent the longitude and latitude of the j-th square respectively, and Q, K, and V represent the two-dimensional matrices of the same first window.

[0062] Then, S3 is executed, subjecting all squares in the entire first panoramic image to a sliding transformation that maintains continuity across the first panoramic image's boundaries, resulting in a new square arrangement. S2 is repeated, and the third eigenvector for each square is obtained based on its second eigenvector. In other words, after the sliding transformation is completed, the square features undergo another in-window self-attention operation based on the new window. After the self-attention operation is completed, the squares are restored to their original arrangement.

[0063] In this embodiment, the step of subjecting all squares of the entire first panoramic image to a sliding transformation that supports the continuity of the boundaries of the first panoramic image specifically includes:

[0064] S31, circularly sliding the first panoramic image as a whole to the left by a distance of half the window size;

[0065] S32: vertically cut the first panoramic image in half, and rotate the right half 180° counterclockwise around the upper right corner of the left half;

[0066] S33: Circularly slide the first panoramic image as a whole downward by a distance of half the window size.

[0067] In this embodiment, the S2-S3 can be cycled multiple times, which can be determined specifically according to the requirements of downstream tasks, and this application does not impose any restrictions on this.

[0068] Then execute S4, copy the first panoramic image once, rotate the copied first panoramic image by 90° in pitch angle to obtain a second panoramic image, transform the first window in the first panoramic image into the second window in the second panoramic image, perform a cross-attention operation on the second window in the second panoramic image and the first window in the first panoramic image, and obtain the fourth eigenvector of each square corresponding to the original first panoramic image based on the third eigenvector of each square.

[0069] Specifically, the squares of the first panoramic image are copied once, and then the original image squares are divided into windows. The copied squares undergo a 90° rotation of the panoramic image's pitch angle. Based on the center position of each window in the first panoramic image, a corresponding window needs to be sampled in the rotated square. The center of the new window corresponds one-to-one with the center of the original window. Then, a cross-attention operation is performed for each pair of corresponding windows from the first and second panoramic images. This cross-attention operation is similar to the attention function, but here the query Q and data V come from the windows in the first panoramic image, while the key value K comes from the window in the second panoramic image.

[0070] The rotation operation can be explained as follows: for any point P(u,v) in the first panoramic image, assume that after the North Pole of the first panoramic image is rotated from P0(0,-0.5π) to the target position P1(u1,v1), P(u,v) will be rotated to P′(u′,v′). The pitch angle rotation operation is expressed as follows:

[0071] Sph(P(u a ,v a )):[sin(u a )cos(v a ),cos(u a )cos(v a )],-sin(v a )

[0072]

[0073]

[0074]

[0075] Among them, ":" means "definition", Represents the vector product, the Angle function gives and The positive direction (i.e. counterclockwise) of the angle is given by Given.

[0076] In this embodiment, for a rotation with a pitch angle of 180°, the target position P1 (u1, v1) should be set to (-π, 0).

[0077] In this embodiment, the steps S2-S4 may be cycled multiple times, which may be determined specifically based on the requirements of downstream tasks, and this application does not impose any restrictions on this.

[0078] Finally, S5 is executed, and based on the fourth eigenvector of each square, the features of every four adjacent squares are merged as a group to obtain the processed features of the first panoramic image. In simple terms, the original squares are divided into new non-overlapping large squares of every 2x2, and the feature vectors of the four small squares in the large squares are arranged into one feature vector, thereby achieving the purpose of reducing the feature size. The size of the feature is expressed as width * height * feature dimension, which can be used for a variety of specific downstream tasks, such as panoramic image classification, panoramic image segmentation, indoor layout prediction, panoramic image target detection, etc. The target loss function of the training is also determined by the downstream task. Furthermore, after S5, the transformer-based panoramic image feature representation learning method also includes:

[0079] S6. Repeat S2-S5 several times to achieve multiple compression of the number of squares, thereby obtaining a smaller feature size, that is, the final feature of the first panoramic image.

[0080] Furthermore, after S6, the transformer-based panoramic image feature representation learning method further includes:

[0081] S7. The final features are used for downstream tasks, such as panoramic image classification, panoramic image segmentation, indoor layout prediction, panoramic image object detection, etc.

[0082] In short, the second square feature obtained by the self-attention operation of S2 will notice the other squares in the first window, but the squares in different first windows cannot exchange information. S3 slides the squares so that the squares originally belonging to different first windows enter the same window. The third square feature obtained by the self-attention operation will notice the other squares outside the original first window (i.e., in the new window after the sliding transformation). S4 rotates the pitch angle by 90°, and the fourth square feature obtained by the cross-attention operation will notice the square features under other perspectives. Therefore, the order of the three steps S2, S3, and S4 can be swapped, and this application does not impose specific restrictions on this, but they must be executed before S5.

[0083] The whole method can achieve compatibility with two-dimensional plane images without retraining parameters, just make B ij ←β ij , and then transform the window sliding mode in the panoramic sliding self-attention operation into a simple horizontal and vertical sliding, with a sliding distance of half a window, and then transform the pitch attention operation into a general self-attention operation. The model after the above changes can be called the plane mode model, and the original model that extracts panoramic image features is called the panoramic mode model. When training the model, you need to first train the plane mode model with a two-dimensional plane image, and then fix β ij And train the model for panoramic mode.

[0084] In summary, the embodiments of the present invention provide a transformer-based panoramic image feature representation learning method, which overcomes the problem of boundary discontinuity in panoramic images by adopting a sliding transformation that supports the continuity of panoramic image boundaries, and overcomes the problem of spatial distortion in panoramic images by position encoding based on Cartesian coordinates, relative position offset based on great circle distance, and an important pitch attention mechanism, thereby extracting high-quality panoramic image features.

[0085] The above description is merely a description of preferred embodiments of the present invention and does not limit the scope of the present invention. Any changes or modifications made by persons skilled in the art based on the above disclosure are within the scope of protection of the present invention. Obviously, various modifications and variations may be made by persons skilled in the art without departing from the spirit and scope of the present invention. Thus, if such modifications and variations fall within the scope of the present invention and its equivalents, the present invention is intended to include such modifications and variations.

Claims

1. A transformer-based panoramic image feature representation learning method, characterized by: include: S1. Divide the first panoramic image into several squares, extract features of each square using a multi-layer deep convolutional neural network, and assign a Cartesian coordinate-based position code to each square feature to obtain a first feature vector of each square; S2. Divide the first panoramic image into a plurality of first windows based on the squares, perform a local self-attention operation on the first eigenvectors of all squares in each first window, and obtain a second eigenvector of each square; S3, subjecting all squares of the entire first panoramic image to a sliding transformation that supports the continuity of the boundaries of the first panoramic image to obtain a new square arrangement, repeating S2, and obtaining a third eigenvector of each square based on the second eigenvector of each square; S4. Copy the first panoramic image once, rotate the copied first panoramic image by 90° in pitch to obtain a second panoramic image, transform the first window in the first panoramic image into the second window in the second panoramic image, perform a cross-attention operation on the second window in the second panoramic image and the first window in the first panoramic image, and obtain fourth eigenvectors corresponding to the respective squares in the original first panoramic image based on the third eigenvectors of the respective squares; S5. Based on the fourth eigenvector of each square, the features of every four adjacent squares are combined as a group to obtain processed features of the first panoramic image.

2. The transformer-based panoramic image feature representation learning method according to claim 1, characterized in that: In S1, for each square, its center is represented by longitude u∈[-π,π] and latitude v∈[-0.5π,0.5π], and the Cartesian coordinates centered on the sphere center are as follows: x=sin(u)cos(v) y=cos(u)cos(v) z=-sin(v) The five scalars x, y, z, u, and v are combined into a first vector, which is then encoded through a fully connected neural network layer to obtain a second vector with the same feature dimension. The first and second vectors are added together to obtain the first feature vector of each grid.

3. The transformer-based panoramic image feature representation learning method according to claim 1, characterized in that: After the first panoramic image is divided into several first windows based on the squares, the squares in the first window will be flattened into a row, thereby converting the first eigenvector of the square into a two-dimensional matrix corresponding to the first window, and the two-dimensional matrix is ​​expressed as the number of squares * feature dimension.

4. The transformer-based panoramic image feature representation learning method according to claim 3, characterized in that: In the self-attention operation of S2, each pair of query Q i and key value k j After the dot product operation, a relative position offset based on the great circle distance will be added, where Q i represents the i-th row of the two-dimensional matrix, k j represents the j-th column of the two-dimensional matrix.

5. The transformer-based panoramic image feature representation learning method according to claim 4, characterized in that: Assume that the self-attention operation is a function called attention, and the attention function is expressed as follows: Where B ij represents the positional relationship between the i-th square and the j-th square, α ij represents the offset of the first panoramic image, β ij Indicates the offset of the plane image, u i and v i Represents the longitude and latitude of the i-th square, u j and v j They represent the longitude and latitude of the j-th square respectively, and Q, K, and V represent the two-dimensional matrices of the same first window.

6. The transformer-based panoramic image feature representation learning method according to claim 1, characterized in that: The step of subjecting all squares of the entire first panoramic image to a sliding transformation that supports the continuity of the boundaries of the first panoramic image specifically includes: S31, circularly sliding the first panoramic image as a whole to the left by a distance of half the window size; S32: vertically cut the first panoramic image in half, and rotate the right half 180° counterclockwise around the upper right corner of the left half; S33: Circularly slide the first panoramic image as a whole downward by a distance of half the window size.

7. The transformer-based panoramic image feature representation learning method according to claim 6, characterized in that: In S3, after obtaining the third eigenvector of each square, the squares are restored to their original order.

8. The transformer-based panoramic image feature representation learning method according to claim 1, characterized in that: In S4, for any point P(u, v) in the first panoramic image, it is assumed that after the North Pole of the first panoramic image is rotated from P0(0, -0.5π) to the target position P1(u1, v1), P(u, v) will be rotated to P′(u′, v′). The pitch angle rotation operation is expressed by the following formula: Sph(P(u a ,v a )):[sin(u a )cos(v a ),cos(u a )cos(v a )],-sin(v a ) Among them, ":" means "definition", Represents the vector product, the Angle function gives and The positive direction of the angle is determined by Given.

9. The transformer-based panoramic image feature representation learning method according to claim 1, characterized in that: After S5, the transformer-based panoramic image feature representation learning method further includes: S6. Repeat S2-S5 several times to obtain the final features of the first panoramic image.

10. The transformer-based panoramic image feature representation learning method according to claim 9, characterized in that: After S6, the transformer-based panoramic image feature representation learning method further includes: S7. Use the final features obtained for downstream tasks.

Images