An isometric image construction method suitable for standard CNNs network processing

Abstract

An isometric image construction method suitable for standard CNN network processing, which is characterized by a large number of perspective images and a variety of types, is used in the semantic segmentation of panoramic images, the two-dimensional plane representation form of the panoramic image is called isometric image, the conversion relationship between spherical image and tangent plane image and the conversion relationship between spherical image and isometric image are used to realize the conversion of perspective image to isometric image, the best projection position of the perspective image data set converted into the isometric image data set is given by taking the isometric image semantic segmentation as an example, and compared with other methods of processing isometric image, it is verified that the method has better performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of panoramic image semantic segmentation, and particularly relates to an equiangular image construction method suitable for standard CNNs network processing. BACKGROUND

[0002] At present, convolutional neural networks (CNNs) have excellent performance in image classification, image segmentation, target recognition and other fields. Especially, a CNNs network pre-training model based on a large-scale labeled data set can be well used as an initial model for solving new problems and can be fine-tuned on a new data set. The data object processed by CNNs is usually a perspective image (i.e. a common two-dimensional plane image). The perspective image has the characteristic that the same object has the same shape at different positions in the perspective image, i.e. translational invariance. Therefore, the convolution kernel of CNNs adopts a regular square. However, the equiangular image does not have the translational invariance feature. To solve this problem, some researches have proposed a method of twisting the convolution kernel, i.e. adaptively adjusting the shape of the convolution kernel according to the shape twisting degree of the object. Coors et al. [Learning Spherical Representations for Detection and Classification in Omnidirectional Images [C]. 15th European Conference on Computer Vision (ECCV)] and Tateno et al. [Distortion-Aware Convolutional Filters for Dense Prediction in Panoramic Images [C]. 15th European Conference on Computer Vision (ECCV)] change the shape of the convolution kernel through spherical projection: the closer to the middle position of the equiangular image, the closer the shape of the convolution kernel to the square; the closer to the upper and lower ends of the equiangular image, the left and right boundaries of the convolution kernel are inclined to both sides to adapt to the change of the shape of the object in the region. The premise of using the twisted convolution kernel is supervised learning, i.e. assuming that there is a labeled equiangular image data set.

[0003] With the popularity of 360° cameras, 360° spherical images / videos are accumulating, but due to the high cost of annotation, there is still a lack of labeled equirectangular image data. To solve the above problem, some researches use domain adaptation methods to process equirectangular images. Su et al. [Learning Spherical Convolution for Fast Features from 360 degrees Imagery[C]. 31st Annual Conference on Neural Information Processing Systems (NIPS)] input the cut-plane dataset of perspective images and equirectangular images into the pre-trained CNNs network model and the CNNs network model to be solved respectively, and constantly optimize the parameters of the CNNs network to be solved, so that the output of the CNNs network to be solved is close to the output of the pre-trained CNNs network. Although this method does not need labeled equirectangular images as a training set, the parameter amount of the model is very large (up to GB level), which is difficult to train and use. The follow-up work of Su et al. [Kernel Transformer Networks for Compact Spherical Convolution[C]. IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)] optimizes the method, reducing the parameter amount of the model, but the performance of the improved method is determined by the pre-trained model, and when the pre-trained model is not good, the effect of the model on equirectangular images will be affected. Ma et al. [Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation with Attention-Augmented Context Exchange[C]. IEEE Intelligent Transportation Systems Conference (ITSC)] proposed an equirectangular image semantic segmentation model based on generative adversarial network. The idea of this model is similar to domain adaptation. This model inputs the perspective image (with label) into a network (generator G) for segmentation to generate the segmentation result, and inputs the equirectangular image into the network to get the segmentation result. The goal of the model is to make the two segmentation results as close as possible. To this end, the segmentation results are passed through the discriminator D to determine whether the input is from the perspective image or the equirectangular image. Through the backward propagation of segmentation loss and adversarial loss, the generator network parameters are optimized, and the generator is optimized to be able to generate good segmentation results for equirectangular images as perspective images.This method also does not require labeled equirectangular images, but this study is for the middle part of the equirectangular image with a lighter degree of distortion, and there is no study on the upper and lower ends of the severely distorted image. SUMMARY

[0004] In view of the problem that it is difficult to effectively learn and extract the shape features of severely distorted objects in equirectangular images due to the lack of large-scale labeled 360° spherical image datasets, the present application seeks to design a processing method for converting existing perspective image datasets into Tagged equirectangular image datasets, so that standard CNNs models can effectively process equirectangular images, especially the upper and lower ends with severely distorted object shape features.

[0005] The present application aims to overcome the shortcomings of the prior art and seeks to design an equirectangular image construction method suitable for standard CNNs network processing, which specifically includes the following steps:

[0006] (1) Given the width w and height h of the equirectangular image to be generated, determine the sampling unit length Δ θ and

[0007] (2) For convenience of calculation, adjust the perspective image size to be composed of n×n sampling points, and regard the perspective image as the tangent plane of any point on the unit sphere , r represents any point on the tangent plane, and the coordinates of each sampling point in the perspective image are calculated according to formulas (3)-(6);

[0008] r(0,0)=(0,0) (3)

[0009] r(±i,0)=(±itan(Δθ),0) (4)

[0010]

[0011]

[0012] (3) Select the spherical tangent point position According to the inverse transformation of the spherical center projection (formulas (7)-(10)), map the determined sampling point r(i,j) in the perspective image to the spherical surface to obtain the spherical coordinates of the corresponding mapping point ;

[0013]

[0014]

[0015]

[0016] v = tan -1 p (10)

[0017] (4) According to the coordinate conversion relationship between the spherical image and the equirectangular image in the following formula, the spherical coordinates of the mapping point are converted into the Cartesian coordinates of the corresponding point p(x, y) in the equirectangular image, and the corresponding equirectangular image is generated.

[0018]

[0019]

[0020] Compared with the generative adversarial network method (the model selects UNet, UNet-P2PDA), the overall segmentation of the method is better, and it also shows that the generative adversarial network method is not good for processing the upper and lower sides of the panoramic image with serious distortion; compared with the conventional image enhancement method using noise addition and rotation, although the method also processes the data set, it has stronger pertinence to the distortion elements, and the overall segmentation result is better than the conventional image enhancement method; the method of Tateno et al. (Tangent plane method) is also experimented, which modifies the convolution kernel like the method of Coors et al., and the modified convolution kernel has better perception ability to distortion, but the error of data sampling is also larger, since the original author mainly studies image classification and depth prediction, the present application is used in semantic segmentation, and after modifying the overall convolution kernel of the model, it is found through comparison with the present method that the processing effect is not good, although the modification of the convolution kernel will cause certain sampling error, but it is still a direction for further research, and the applicability of the convolution kernel in the model will be studied in future research.

[0021] Compared with the prior art, the present application takes the equirectangular image semantic segmentation as an example, and gives the best projection position of converting the perspective image data set into the equirectangular image data set, and verifies that the method has better performance compared with other methods of processing equirectangular images. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a panoramic image.

[0023] Figure 2 is Figure 1 an equirectangular image corresponding to the panoramic image.

[0024] Figure 3 is a unit sphere.

[0025] Figure 4 is Figure 3 a two-dimensional plane (represented by a spherical coordinate system) corresponding to a sampling point on the unit sphere.

[0026] Figure 5 For Figure 3 The two-dimensional plane corresponding to the sampling point on the unit sphere (Cartesian coordinate system representation).

[0027] Figure 6 For the tangent plane I at the tangent point (0, 0) r (a) and the projection area I corresponding to the tangent plane s (b).

[0028] Figure 7 For I r The corresponding relationship between the upper point r(i, j) and its projection point s(i, j) on I s (a) is The interval value in the (vertical) direction, and (b) the interval value in the θ (horizontal) direction.

[0029] Figure 8 For the tangent plane at the arbitrary point on the sphere.

[0030] Figure 9 For the conversion of the perspective image to the equirectangular image.

[0031] Figure 10 For the construction process of the test set. DETAILED DESCRIPTION

[0032] The application will be further described below by examples.

[0033] Example 1

[0034] By virtue of the characteristics of a large number of perspective images and rich types, the perspective images are used in the semantic segmentation of panoramic images. In this embodiment, the two-dimensional plane representation form of the panoramic image is called the equirectangular image. The conversion relationship between the spherical image and the tangent plane image and the conversion relationship between the spherical image and the equirectangular image are used to realize the conversion of the perspective image to the equirectangular image. The specific conversion mode is as follows:

[0035] (1) Establish the coordinate conversion relationship between the spherical image and the equirectangular image:

[0036] As shown in Figure 3 , the viewpoint of the spherical image is at the center of the sphere. Assuming that a unit sphere is s, any point on the sphere is uniquely determined by the azimuth angle θ ∈ [-π, π] and the zenith angle . The equirectangular projection is an equal-interval (Δ θ and ) sampling of the spherical image, that is, continuously moving along the horizontal direction according to the interval angle Δ θ ​ Or along the vertical direction Continuous movement of interval angle Each movement yields a sampling point on the sphere. These sampling points are then uniformly mapped onto a two-dimensional plane, resulting in a two-dimensional plane representing the sampling point on the unit sphere in spherical coordinates, as shown below. Figure 4 As shown, let the width w and height h of the two-dimensional plane be... This represents the length of each unit in the horizontal direction on a two-dimensional plane. This represents the length of each unit in the vertical direction (in this embodiment, a unit sphere is used, where the arc length and radian value are equal, Δ). θ and (This also represents equally spaced angles and equally spaced lengths). When performing image processing on a two-dimensional plane, the spherical coordinate system is usually converted to a Cartesian coordinate system, and formulas (1) and (2) are used to represent any point in the spherical coordinate system. Convert the point p(x,y) to Cartesian coordinates, such as Figure 5 As shown.

[0037]

[0038]

[0039] (2) Establish the relationship between the spherical image and the tangent plane:

[0040] For any point on the surface of a sphere, a tangent plane can be constructed. The point where the tangent plane intersects the sphere is called the point of tangency. For example... Figure 6 As shown, with the sphere (0,0) as the tangent point, a tangent plane consisting of n×n sampling points with tangent point (0,0) is constructed, denoted as I. r I r Projecting the sphere onto the surface through the center of the sphere forms a projection region consisting of n×n projection points, denoted as I. s I r Upsampling point r(i,j)( Figure 6 The orange dot in a) and its projection on I s The projection point s(i,j)( on) Figure 6 There is a one-to-one correspondence between the black dots in a.

[0041] like Figure 7 As shown, tangent plane I r The interval between two adjacent sampling points in the horizontal direction is tan(Δθ), and the interval between two adjacent sampling points in the vertical direction is... Then, in the tangent plane I rThe Cartesian coordinates of any sampling point r(i,j) are calculated as shown in equations (3)-(6), where the left side of the equation is the sampling point, and the right side of the equation is the coordinate of the corresponding sampling point in the Cartesian coordinate system, wherein, (i,j) represents the position of the sampling point in the tangent plane, and r(0,0) represents the sampling point at the center of the tangent plane (i.e., the tangent point), and r(0,1) represents the first sampling point above the tangent point.

[0042] r(0,0) = (0, 0) (3)

[0043] r(±i,0) = (±itan(Δθ), 0) (4)

[0044]

[0045]

[0046] The shape of the tangent plane at the tangent point (0,0) is kept unchanged, and the tangent plane can be moved on the surface of the sphere, and the tangent point is arbitrarily selected on the sphere The points on the tangent plane can be mapped onto the sphere by the inverse transformation of the spherical center projection, as shown in equation (7). Figure 8

[0047] Let the coordinates of the sampling points on the tangent plane in the Cartesian coordinate system be r = (i,j). Since the relative positions of the sampling points on the tangent plane do not change during the movement, the coordinates r are obtained from equations (3)-(6), and the azimuth angle of the corresponding point mapped onto the sphere is θ(i,j), and the zenith angle is θ(i,j). The calculation formulas are as follows:

[0048]

[0049]

[0050]

[0051] v = tan -1 ρ (10)

[0052] where θ and are the coordinates of the selected tangent point.

[0053] (3) Conversion of a perspective image to an isometric image:

[0054] Based on the principles of sections (1) and (2), the conversion of a perspective image to an isometric image can be achieved by the following steps.

[0055] (301) Given the width w and the height h of the isometric image to be generated, determine the sampling unit length Δ θ and​

[0056] (302) For the convenience of calculation, the size of the perspective image is adjusted to be composed of n x n sampling points, and the perspective image is taken as a unit sphere The coordinates of each sampling point in the perspective image are calculated according to the formula (3)-(6) according to the tangent plane of the above point;

[0057] (303) Selection of the spherical tangent point position According to the inverse transformation of the gnomonic projection (formula (7)-(10)), the determined sampling point r(i,j) in the perspective image is mapped to the sphere to obtain the spherical coordinates of the corresponding mapping point ;

[0058] (304) According to the coordinate conversion relationship (formula (1)-(2)) between the spherical image and the equirectangular image, the spherical coordinates of the mapping point are converted into the Cartesian coordinates of the corresponding point p(x,y) in the equirectangular image to generate the corresponding equirectangular image.

[0059] As shown in Figure 9 , the yellow point s is the selected tangent point, and the red point r=(r x ,r y ) is an arbitrary point on the I r image, and the corresponding mapping point on the sphere is the black point Then according to the equirectangular projection, the point on the sphere is mapped to the red point p(x,y) in the equirectangular image.

[0060] The selection of the tangent point determines the degree of distortion of the shape of the object in the equirectangular image converted from the perspective image. When is closer to π / 2 or -π / 2 (the two poles of the sphere), the shape of the object in the projected equirectangular image is more distorted, and vice versa, when is closer to 0, the shape of the object is closer to the shape in the perspective image. The value of θ determines the horizontal position of the projection area in the equirectangular image, and has no effect on the shape of the object, so θ can be set to 0 during conversion.

[0061] The sampling points in the embodiment can be understood as pixel points.

[0062] Embodiment 2

[0063] The existing perspective image dataset is converted into a labeled isometric image dataset by the method in Embodiment 1, so that the standard CNNs model can effectively process the isometric image (especially the upper and lower ends of the object shape feature distortion), and the best projection position of the perspective image dataset converted into the isometric image dataset is given by taking the isometric image semantic segmentation as an example. Compared with other methods of processing isometric images, it is verified that the method has better performance.

[0064] Perspective image dataset introduction:

[0065] CityScapes is a perspective image dataset containing different city street views, which has a total of 34 categories, and 6 categories (road, building, vegetation, sky, car, and pedestrian) are selected for the experiment. CamVid is a perspective image dataset containing different city street view images, extracted from video frames, including 32 categories, and the category selection is consistent with CityScapes.

[0066] Experimental scheme:

[0067] The above two perspective image datasets of CityScapes and CamVid are converted into isometric images by using the method in Embodiment 1, and the Omni-CityScapes dataset and the Omni-CamVid dataset are obtained.

[0068] Two experimental schemes are designed, CityScapes is used as the training set, and the Omni-CamVid dataset is used as the test set; CamVid is used as the training set, and the Omni-CityScapes dataset is used as the test set. When constructing the test set, the projection value is set to π / 2, so that the object shape in the test set is distorted enough; when constructing the training set, the projection value is selected in the range of (0, π / 2) to find the best value, so that the standard CNNs model has the best learning ability on the isometric image projected by the value. The image resolution used in training is 224x224 pixels, the number of test sets is 1000, and the number of training sets is 700.

[0069] Dataset construction:

[0070] Since the training set is directly converted from perspective images by spherical projection, in order to make the image space layout of the test set inconsistent with the training set, a cropping process is added before the test set is generated by spherical projection conversion using the perspective image dataset. The cropping principle is to ensure that each category can be projected to different positions on the equirectangular image projection area after cropping and spherical conversion, so that the test set can contain different positions of distortion, and at the same time, to ensure that the projected image is always on the divided equirectangular image part, the part of the projected test set that exceeds is cropped, as shown in Figure 10

[0071] Experiments are conducted using the CityScapes and CamVid datasets and multiple standard CNN semantic segmentation models. The experimental objectives are (1) to give the best projection value of converting perspective images into equirectangular images to ensure that the standard CNN semantic segmentation model can better learn the shape features of objects with severe distortion; and (2) to verify that the proposed equirectangular image construction method can effectively improve the performance of standard CNNs in semantic segmentation of equirectangular images by comparison with other methods.

[0072] Projection value determination:

[0073] (1) Different datasets, same CNN network structure

[0074] The projection values of the training set are set to π / 16, 2π / 16, 3π / 16, 4π / 16, 5π / 16, 6π / 16, 7π / 16, and 8π / 16, respectively. The semantic segmentation model uses the UNet model, and the VGG16 model pre-trained on the ImageNet dataset is used as the encoder of UNet. Considering the distortion of object shapes in equirectangular images, the encoder / decoder of the UNet model uses distorted convolution kernels and ordinary convolution kernels, respectively, to analyze the influence of the two different convolution kernels on the performance of the semantic segmentation model. The number of training rounds used in the experiment is 40 rounds, and the IoU index commonly used in semantic segmentation is used as the test index. The learning rate is 1×10^(-4) before 25 rounds, and then the learning rate is adjusted to 1×10^(-5).

[0075] The CamVid dataset is projected to different positions to construct the training set, and the UNet model with distorted convolution kernels is used for training. After training, the IoU results on the test set Omni-CityScapes are shown in Table 1. When the perspective image is projected to the spherical ​When the value is 6π / 16, the overall segmentation effect of the model is better than that of other projection positions, the average IoU value is 39.76, and the segmentation effect of buildings and sky is the best. The segmentation effect of roads and vegetation is not much different from the best value. Table 2 shows the segmentation effect of the model trained by using the ordinary convolution kernel on the test set. It can be seen that when the projection is 6π / 16, the average IoU value of the six categories is 43.76, which is better than that of other projection positions. By comparing Table 1 and Table 2, it can be found that the IoU indicators of the UNet model using the ordinary convolution kernel at 8 different projection positions are better than those of the UNet model using the twisted convolution kernel.

[0076] Table 1 Segmentation indicators IoU (%) of the UNet model using the twisted convolution kernel when the CamVid dataset is projected to different positions on the plane

[0077]

[0078] Table 2 Segmentation indicators IoU (%) of the UNet model using the ordinary convolution kernel when the CamVid dataset is projected to different positions on the plane

[0079]

[0080] Table 3 Segmentation indicators IoU (%) of the UNet model using the twisted convolution kernel when the CityScapes dataset is projected to different positions on the plane When the value is 6π / 16, the overall segmentation effect of the model for the six categories is the best, the average IoU value is 30.63, and the segmentation effect of roads, buildings and sky is the best compared with other projection positions. Table 4 shows the segmentation effect of the UNet model trained by using the ordinary convolution kernel on the test set. It can be seen that when the projection is 4π / 16, the segmentation effect of buildings and cars is the best, when the projection is 8π / 16, the segmentation effect of roads and vegetation is the best, and when the projection is 6π / 16, the segmentation effect of sky is the best. The average IoU value of the six categories is 31.70, which is better than that of other projection positions. By comparing Table 3 and Table 4, it can be found that the average IoU indicators of the model using the ordinary convolution kernel at 8 different projection positions are better than those of the model using the twisted convolution kernel.

[0081] Table 3 Segmentation indicators IoU (%) of the UNet model using the twisted convolution kernel when the CityScapes dataset is projected to different positions on the plane

[0082]

[0083]

[0084] Table 4 Segmentation index IoU (%) of UNet model with ordinary convolution kernel for different positions of CityScapes dataset projected to a plane

[0085]

[0086] (2) Same dataset, different CNNs network structure

[0087] The training set uses the CamVid dataset, the test set uses the Omni-Cityscapes dataset, and the UNet, UNet++, SegNet, PSPNet, and DeepLab v3+ networks are used to train the training set, and the ordinary convolution kernel is used. The experimental results are shown in Tables 5-9. It can be seen that when the average IoU of UNet, Segnet and PSPNet network models is better than that of other values, when using UNet++ and DeepLabv3+ the average IoU of 5π / 16 is the best, but the difference between the average IoU of 6π / 16 is very small, which is 0.03 and 0.33 respectively, which shows that even if part of the model does not reach the optimal value at 6π / 16 projection position, but for the results projected to 6π / 16, the difference between the two is very small, so as in part (1), for different semantic segmentation models, 6π / 16 can be approximated as the actual optimal projection position.

[0088] Table 5 Segmentation index IoU (%) of UNet network when the backbone network is VGG16 and ResNet34

[0089]

[0090]

[0091] Table 6 Segmentation index IoU (%) of UNet++ network when the backbone network is VGG16 and ResNet34

[0092]

[0093] Table 7 Segmentation index IoU (%) of SegNet network when the backbone network is VGG16

[0094]

[0095] Table 8 Segmentation index IoU (%) of PSPNet network when the backbone network is VGG16 and ResNet34

[0096]

[0097] Table 9 Segmentation index IoU (%) of DeepLab v3+ network with ResNet34 as the backbone network

[0098] Table 9 Segmentation index IoU (%) of DeepLab v3+ network with ResNet34 as the backbone network

[0099]

[0100] (3) Method comparison

[0101] Three different methods are used for comparison with the method of the present application, which are supervised learning, unsupervised learning and data enhancement method. Except for method 1, the training set uses CamVid dataset, and the test set uses Omni-Cityscapes dataset, which are 700 and 1000 respectively, and the semantic segmentation model uses UNet model.

[0102] 1. Cut-plane image method: ordinary convolution kernel is used for training, and twisted convolution kernel is used for testing. The equirectangular image in the Omni-Cityscapes dataset is used to generate the cut-plane image by using the spherical projection, and the cut-plane image is used as the input of the UNet network model for training. The ordinary convolution kernel weight of the UNet network is copied to the twisted convolution kernel for segmentation of the equirectangular image during testing. The Omni-Cityscapes dataset has 700 images during training. Since Omni-Cityscapes can be regarded as a labeled equirectangular image dataset, it is supervised learning.

[0103] 2. UNet-P2PDA: This method uses an adversarial generative network model, which simultaneously receives perspective images and equirectangular images, so that the closer the segmentation results of the two are, the better. Since no equirectangular image is needed for training, it is unsupervised learning.

[0104] 3. Image enhancement: This method crops, rotates up, down, left and right, etc. on the perspective image to increase the number of datasets to enhance the learning ability of the UNet model.

[0105] When the method of the present application is used, ordinary convolution kernel is used for training and testing. The experimental results of the method compared with the above three methods are shown in Table 10.

[0106] Table 10 Comparison of the method of the present application with other methods (IoU)

[0107]

Claims

1. A method for constructing isometric images suitable for standard CNN network processing, characterized in that, Includes the following steps: (1) Given the width w and height h of the isometric image to be generated, determine the sampling unit length Δ in the horizontal and vertical directions of the isometric image. θ and (2) For ease of calculation, the size of the perspective image is adjusted to consist of n×n sampling points, and the perspective image is used as the unit sphere. The tangent plane at any point is used to calculate the coordinates of each sampling point in the perspective image according to formulas (3)-(6); r(0,0)＝(0,0) (3) r(±i,0)=(±itan(Δθ),0) (4) (3) Select the tangent point of the sphere Based on the inverse transformation of the sphere projection (Equations (7)-(10)), the determined sampling points r(i,j) in the perspective image are mapped onto the sphere to obtain the corresponding mapped points. spherical coordinates; v}tan -1 ρ (10) (4) Based on the coordinate transformation relationship between spherical and isometric images in the formula below, convert the spherical coordinates of the mapped points... Convert the coordinates of the point p(x,y) in the isoangular image to Cartesian coordinates, and generate the corresponding isoangular image.

Images