Rotated convolution method for top-view fisheye vision task

Abstract

The application discloses a rotation convolution method for top-view fisheye vision tasks, when a mask required for calculating rotation convolution is obtained, different direction convolution kernels are applied to different regions of a top-view fisheye image, and when a convolution operation is performed, input feature maps are respectively subjected to center-restricted deformable convolution with different rotation state convolution kernels; the application proposes a rotation convolution kernel based on rotation isomorphism of a convolution neural network, applies the rotated convolution kernel to different positions in the top-view fisheye image, proposes a center-restricted deformable convolution and applies the center-restricted deformable convolution to the rotation convolution, so that the convolution kernel can be adaptively rotated and deformed, the ability of the convolution kernel to extract similar features from objects with different rotation states in the top-view fisheye image is improved, and the accuracy of the convolution neural network in top-view fisheye vision detection and segmentation tasks is greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a rotational convolution method for top-view fisheye vision tasks. Background Technology

[0002] Top-view fisheye cameras are the most cost-effective devices for capturing 360-degree video and have a wide range of applications in visual surveillance. Unlike traditional perspective cameras, which sample the field of view of a 3D scene projected onto a 2D plane, top-view fisheye cameras capture an omnidirectional view around their optical center. Therefore, top-view fisheye images provide more spatial information than traditional perspective images.

[0003] Top-view fisheye cameras, by capturing an omnidirectional view, present greater challenges for performing visual tasks compared to traditional images. There are two main reasons for this: First, traditional images show objects with the same relative positions as in the real world, while objects in top-view fisheye images rotate around the optical axis, and convolutional kernels do not possess rotational equivariance. Second, to capture a larger field of view, more distortion is introduced into top-view fisheye images, which is more severe at the edges. Therefore, the accuracy of ordinary convolutional neural networks is significantly reduced when applied to top-view fisheye vision tasks. Summary of the Invention

[0004] This invention addresses the problems of existing convolution kernels lacking rotational equivariance and introducing more distortion into images. It provides a rotational convolution method for top-view fisheye vision tasks. To address image rotation, this invention proposes a rotational convolution kernel based on the rotational equivariance of convolutional neural networks and applies the rotated kernel to different positions in the top-view fisheye image. To address image distortion, this invention proposes a center-constrained deformable convolution and applies it to rotational convolution. This convolution method allows the convolution kernel to adaptively rotate and deform, improving its ability to extract similar features from objects with different rotation states in the top-view fisheye image, and significantly enhancing the accuracy of convolutional neural networks in top-view fisheye vision detection and segmentation tasks.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is: a rotational convolution method for top-view fisheye vision tasks. When calculating the mask required for rotational convolution, convolution kernels in different directions are applied to different regions of the top-view fisheye image. When performing convolution operations, the input feature map is subjected to center-constrained deformable convolution with convolution kernels in different rotation states.

[0006] As a further improvement to the present invention, the method of the present invention includes the following steps:

[0007] S1, Rotation Convolution Input Image Acquisition: Extract features from the top-view fisheye image and input them into a rotation convolution;

[0008] S2, Mask Calculation: Calculate the mask required for rotational convolution. The mask is used to apply convolution kernels in different directions to different regions of the top-view fisheye image. The generated mask size is the same as the original image size. Downsampling is required when using features at different scales.

[0009] S3, Convolutional Kernel Rotation and Stitching: The convolutional kernel is copied into 4 copies, and the 4 copies of the convolutional kernel are rotated counterclockwise by 0°, 90°, 180° and 270° respectively. The 4 rotated convolutional kernels are stitched together along the output feature dimension.

[0010] S4, Center-Constrained Deformable Convolution: The rotated convolution kernel and the input features are subjected to center-constrained deformable convolution;

[0011] S5, Feature Map Output: Perform a convolution operation on the deformable convolution with center constraint in step S4. The output result is decomposed into 4 parts along the feature dimension, and then multiplied by the corresponding mask calculated in step S2 to obtain the Cartesian product. Finally, add the 4 Cartesian products to obtain the final result of the rotational convolution, and output the feature map.

[0012] Compared to existing technologies, the main approach of existing technologies is group equivariant convolution, which expands the convolution kernel into a rotation group, allowing each kernel in the rotation group to convolve with the input image to generate an output feature map. This method adds new dimensions to the feature map, consuming more data. Furthermore, current methods for optimizing the rotation equivariance of convolutional neural networks are mainly based on randomly rotated images, meaning that the perceptual targets within the image rotate according to random patterns. However, perceptual targets in top-view fisheye images follow fixed patterns, therefore group equivariant convolution cannot optimally adapt to the rotation patterns of top-view fisheye images. This invention addresses the rotation patterns of top-view fisheye images by proposing a rotational convolution kernel based on the rotation equivariance of convolutional neural networks, and applying the rotated kernel to different locations within the top-view fisheye image. To address image distortion, this invention proposes a center-constrained deformable convolution and applies it to the rotational convolution. This convolution method allows the kernel to adaptively rotate and deform, improving the kernel's ability to extract similar features from objects with different rotation states in top-view fisheye images, significantly improving the accuracy of convolutional neural networks in top-view fisheye visual detection and segmentation tasks. Attached Figure Description

[0013] Figure 1 This is a flowchart of the steps of the rotational convolution method for top-view fisheye vision tasks of the present invention;

[0014] Figure 2 This is a schematic diagram of step S2 of embodiment 1 of the present invention for generating the rotational convolution mask;

[0015] Figure 3This is a diagram showing the correspondence between the internal parameters of the convolution kernel during kernel rotation in step S3 of embodiment 1 of the present invention.

[0016] Figure 4 This is a comparative diagram of center-constrained deformable convolution and ordinary convolution in the method of this invention;

[0017] Figure 5 This is a schematic diagram of the rotational convolution principle in step S5 of embodiment 1 of the present invention. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention.

[0019] Example 1

[0020] Rotational convolution methods for top-view fisheye vision tasks, such as Figure 1 As shown, it includes the following steps:

[0021] Step S1: Extract the features from the top-view fisheye image and input them into a rotational convolution.

[0022] The original top-view fisheye image is processed through multiple layers of ordinary convolution to obtain the input feature map for rotational convolution. In convolutional neural networks, lower-level features contain more detailed information, while upper-level features contain more semantic information; therefore, rotational convolution is applied to the upper-level features. The upper-level features are then input into the rotational convolution. After the rotational convolution is complete, the result can be fed into the next rotational convolution or decoded through ordinary convolution to obtain the result of a detection or segmentation task. Specifically, the aspect ratio of the top-view fisheye image used for rotational convolution is 1:1, and the center of the imaging region roughly coincides with the center of the image.

[0023] Step S2: Calculate the mask required for rotational convolution.

[0024] The mask is used to apply convolutional kernels with different orientations to different regions of a top-view fisheye image. Specifically, in the top-view fisheye image, a polar coordinate system is established with the image center as the origin and the rightward direction as the positive direction. Objects in the upper region are generally oriented upwards, so a 90° convolutional kernel is primarily applied to this region; while objects in the right region are generally oriented to the right, so a 0° convolutional kernel is primarily applied to this region, and so on. The generated mask size is the same as the original image size, and downsampling is required when using it for features at different scales. Taking the mask used for the upper region as an example, such as... Figure 2 As shown, a 90° mask for the upper region is generated using the following polar coordinate system formula, with the same resolution as the input image:

[0025]

[0026] Where r is the radius in polar coordinates and θ is the angle in polar coordinates.

[0027] Rotate the 90° mask 90° clockwise to obtain the 0° mask for the right region, rotate the 0° mask 180° counterclockwise to obtain the 180° mask for the left region, and rotate the 0° mask 270° counterclockwise to obtain the 270° mask for the lower region.

[0028] Step S3: Copy the convolution kernel into 4 copies, and rotate the 4 copies of the convolution kernel counterclockwise by 0°, 90°, 180° and 270° respectively.

[0029] Specifically, the input feature map dimensions are [B, C_in, W_in, H_in], where B is the batch dimension of the feature map, C_in is the input channel dimension of the feature map, and W_in and H_in are the width and height of the input feature map, respectively. The convolution kernel dimensions are [C_out, C_in, w, w], where C_out is the output feature dimension and w is the width of the convolution kernel. Taking a 3×3 convolution kernel with w=3 as an example, the rotation is performed according to... Figure 3 The parameter positions shown indicate where data is read; the parameter 'a' is located at the top left corner of the convolution kernel before rotation. 00 After rotating 90 degrees, move to the bottom left corner of the convolution kernel; similarly, move the parameter 'a' to the top right, bottom right, and bottom left corners. 02 a 22 a 20 After rotating 90 degrees, the parameters are moved to the top left, top right, and bottom right corners of the convolution kernel. When rotating 180 degrees, the parameters at the top left, top right, bottom right, and bottom left corners are moved to the bottom right, bottom left, top left, and top right corners, respectively. Other parameters are rotated in a similar manner. The four rotated convolution kernels are then concatenated along the output dimension, resulting in a concatenated convolution kernel with dimensions of [4*C_out,C_in,w,w].

[0030] Step S4: Perform center-constrained deformable convolution on the rotated convolution kernel and the input features.

[0031] like Figure 4 As shown, due to the distortion of fisheye images, the shape of the perceived target changes and image distortion, resulting in poor performance of traditional convolutional neural network structures for target perception in fisheye images. Optimizing convolutional neural networks with deformable convolution is an effective method to adapt deep learning models to image distortion. By learning the bias and weights of the convolution kernel grid positions through additional convolutional layers, and then fixing the center position of the convolution kernel, center-constrained deformable convolution is achieved. Since the convolution kernel and output features are concatenated in the same way, a single convolution operation allows the input feature map to undergo center-constrained deformable convolution with convolution kernels in four different rotation states. The center-constrained deformable convolution is obtained by the following method:

[0032]

[0033] Where p is the position coordinate in the feature map; p k ∈{(-1,-1),(-1,-0),...,(1,0),(1,1)} represents the coordinates of the sampling positions of the convolution kernel grid, where (-1,-1) and (0,0) are the indices of the top-left corner and the center, respectively; k is the number of sampling positions of the convolution kernel grid, and w k The parameters of the convolution kernel at different kernel grid positions; △p k , △m k , representing the offsets of the feature grid sampling positions and coefficients in the deformable convolution, respectively, are learned by a regular convolutional module; x and y represent the input and output features, respectively. To ensure more spatial structure, the center of the deformable convolution kernel is constrained, keeping its position and coefficient offsets constant.

[0034] Step S5: Multiply the four convolution results by the mask and add them bit by bit to obtain the result of the rotational convolution.

[0035] Specifically, the output feature dimension obtained from step S4 is [B, 4*C_out, W, H], where B is the batch dimension of the feature map, C_out is the output channel dimension of the rotation convolution, and W_out and H_out are the width and height of the output feature map, respectively. This feature is decomposed along the feature dimension into four feature maps of dimension [B, C_out, W, H]. The first part is the output feature of the input feature interacting with a 0° convolution kernel, the second part is the output feature of the input feature interacting with a 90° convolution kernel, and so on. Figure 5 As shown, to apply the 0° convolution kernel to the right D region of the top-view fisheye image, the first output feature is calculated as a Cartesian product with the 0° mask generated in step S2. Similarly, the second, third, and fourth output features are calculated as Cartesian products with the 90°, 180°, and 270° masks generated in step S2. Finally, the above four output feature maps are directly added together to obtain a rotated convolution output feature map with dimensions [B, C_out, W, H].

[0036] Example 2

[0037] Taking the top-level fisheye feature with input dimension [64,64,32] and the feature map with output dimension [64,64,16] as an example, the rotational convolution method proposed in this invention is illustrated.

[0038] First, using the rotational convolution mask generation method described in step S2, four masks with dimensions [64, 64] are generated using the polar coordinate formula, as follows: Figure 2As shown, these are masks facing right at 0°, upward at 90°, left at 180°, and downward at 270°, respectively. Then, as described in step S3, the original convolutional kernel with dimensions [32, 16, 3, 3] is copied into four copies. These four copies are then rotated counterclockwise by 0°, 90°, 180°, and 270° respectively using index coordinates. The four convolutional kernels are then stacked together along the output dimension (i.e., the second dimension) to form a convolutional kernel with dimensions [32, 64, 3, 3]. Using a newly obtained convolutional kernel of dimensions [32, 64, 3, 3] and input features of dimensions [64, 64, 32], a deformable convolution with center constraints is performed to obtain an output feature map of dimensions [64, 64, 64]. Then, the output feature map is decomposed along the channel dimension (i.e., the third dimension) into four feature maps of dimensions [64, 64, 16]. The first part is the result of convolving the input features with a 0° convolution kernel, the second part is the result of convolving the input features with a 90° convolution kernel, and so on. Next, the four feature maps of dimensions [64, 64, 16] are multiplied bitwise with four masks of dimensions [64, 64] respectively. Finally, the results of the multiplications are added bitwise to obtain the output feature map of dimensions [64, 64, 16]. This method ensures that the four convolutional kernels with different orientations primarily act on four different regions of the image, achieving rotational convolution.

[0039] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A rotational convolution method for top-view fisheye vision tasks, characterized by: When calculating the mask required for rotational convolution, convolution kernels with different orientations are applied to different regions of the top-view fisheye image. During the convolution operation, the input feature map is subjected to center-constrained deformable convolution with convolution kernels in different rotation states. This includes the following steps: S1, Rotation Convolution Input Image Acquisition: Extract features from the top-view fisheye image and input them into the rotation convolution; S2, Mask Calculation: Calculate the mask required for rotational convolution. The mask is used to apply convolution kernels of different directions to different regions of the top-view fisheye image. The generated mask size is the same as the original image size. Downsampling is required when using features at different scales. S3, Convolutional Kernel Rotation and Stitching: The convolutional kernel is copied into 4 copies, and the 4 copies of the convolutional kernel are rotated counterclockwise by 0°, 90°, 180° and 270° respectively. The 4 rotated convolutional kernels are stitched together along the output feature dimension. S4, Center-Constrained Deformable Convolution: The rotated convolution kernel and the input features are subjected to center-constrained deformable convolution; S5, Feature Map Output: Perform a convolution operation on the deformable convolution with center constraint in step S4. The output result is decomposed into 4 parts along the feature dimension, and then multiplied by the corresponding mask calculated in step S2 to obtain the Cartesian product. Finally, add the 4 Cartesian products to obtain the final result of the rotational convolution, and output the feature map.

2. The rotational convolution method for top-view fisheye vision tasks as described in claim 1, characterized in that: In step S1, the features of the top-view fisheye image are obtained by performing multiple layers of ordinary convolution operations on the original top-view fisheye image. The top-level features are derived from a top-view fisheye image with an aspect ratio of 1:1 and the center of the imaging region coincides with the center of the image.

3. The rotational convolution method for top-view fisheye vision tasks as described in claim 1, characterized in that: In step S3, a polar coordinate system is established in the top-view fisheye image with the image center as the origin and the rightward direction as the positive direction. The polar coordinate formula for the 90° mask in the upper region is: ； Where, r ( () is the radius in polar coordinates. Angles in polar coordinates; The masks for other areas can be obtained by rotating the mask.

4. The rotational convolution method for top-view fisheye vision tasks as described in claim 1, characterized in that: In step S4, the center-constrained deformable convolution is obtained by the following method: ； Where p is the position coordinate in the feature map; Let (-1, -1) and (0, 0) be the coordinates of the sampling positions of the convolution kernel grid, where (-1, -1) and (0, 0) are the indices of the top left corner and the center, respectively; k is the number of sampling positions of the convolution kernel grid. These are the parameters of the convolution kernel at different kernel grid positions; , , respectively, represent the sampling positions of the feature grid and the offsets of the coefficients in deformable convolution; x(˙) and y(p) are the input and output features, respectively.

5. The rotational convolution method for top-view fisheye vision tasks as described in claim 4, characterized in that: When calculating the center-constrained deformable convolution in step S4, the positional offset in the deformable convolution is... and coefficient offset It remains constant.

Images