Spherical convolution-based all-round fisheye image 3D sensing method and related equipment

Through spherical convolution processing of circumferential fish-eye images, the problems of unconsidered fish-eye camera image distortion and circumferential images are solved, and more efficient 3D perception and obstacle recognition are achieved, reducing viewing angle loss and improving recognition accuracy.

CN119090738APending Publication Date: 2024-12-06XI AN JIAOTONG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411243233.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The images acquired by the fisheye camera have severe distortion, resulting in the inability to adopt planar image perception technology, and the prior art has failed to effectively consider the circumferential image, and more fisheye cameras are required to perform viewing angle overlap to reduce viewing angle loss.

Method used

Using the 3D perception method of circumferential fish eye image based on spherical convolution, multiple circumferential fish eye images are sphericalized into panoramic images with equal rectangular projection, and input them into the spherical convolution neural network for isovariant feature extraction, and the spherical convolution processing is used to reduce Loss of viewing angle and improves obstacle recognition accuracy.

Benefits of technology

It greatly reduces viewing angle loss, maximizes image edge characteristics, and improves the accuracy of edge obstacle recognition and vision grid representation of fisheye image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090738A_ABST
    Figure CN119090738A_ABST
Patent Text Reader

Abstract

The invention discloses an all-round fisheye image 3D sensing method based on spherical convolution and related equipment, all-round fisheye images are processed through panoramic stitching and spherical convolution, the visual angle loss is greatly reduced, the image edge features can be kept to the maximum extent by performing equal rectangular panorama projection on a plurality of fisheye images, and the image quality is improved. This is a major pain point problem of a fisheye vision task. And on the other hand, the spherical convolution method enables the convolutional neural network to learn more information from the distortion features of the look-around fisheye image, and inputs the panorama into the constructed spherical convolution network to perform multi-layer feature extraction, thereby improving the accuracy of recognition of the obstacle at the edge of the fisheye image and expression of the long-shot occupied grid.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of fisheye cameras, and specifically relates to a 3D perception method of a surround-view fisheye image based on spherical convolution and related equipment. Background Art

[0002] Fisheye cameras are widely used in road perception tasks due to their wide field of view and high performance in various applications. The ability to capture wide-angle views without the need for multiple traditional cameras makes fisheye cameras a cost-effective solution for real-time 3D perception in dynamic environments. However, the images obtained by fisheye cameras have severe distortion, which makes it impossible to use planar image perception technology. To address this difficult problem, the present invention provides a surround fisheye image processing solution based on panoramic stitching, and uses the panoramic image for obstacle detection and semantic scene reconstruction.

[0003] In CN117455954A "A distortion-optimized multi-target tracking algorithm based on a vehicle-mounted fisheye camera", a distortion-optimized multi-target tracking algorithm based on a vehicle-mounted fisheye camera is proposed, which includes the following steps: 1. Preprocessing the pictures taken by the fisheye camera, the image is distorted and corrected as the input of the target detection algorithm; 2. Initializing parameters, which include a matrix related to the distance; 3. Detecting the target through the yolov5 detector to obtain a detection image with a detection frame; 4. Using the detection image with the detection frame as input to obtain each pedestrian annotation frame and feature map; 5. Filtering out the annotation frames whose confidence is less than the first confidence; 6. Performing non-maximum suppression; 7. Performing trajectory prediction and trajectory update through the Kalman filter; 8. Outputting the ID number of the successfully tracked annotation frame.

[0004] In CN117782102A "A method for fully automatic parking positioning and mapping based on surround vision", the surround view image is first obtained by installing fisheye cameras on the front, rear, left and right of the car, and then a bird's-eye view image is obtained through camera distortion removal, inverse perspective transformation and image stitching operations; then a convolutional neural network is used to extract lane lines and obstacle information in the bird's-eye view image, and it is converted into a point cloud; the IMU and odometer data are integrated to estimate the position and generate a global map; finally, the map is optimized through loop detection to generate a grid map for fully automatic parking.

[0005] The input image in "A Distortion Optimized Multi-Target Tracking Algorithm Based on Vehicle-mounted Fisheye Camera" does not take into account the surround view image, and there are problems in detecting invisible targets under a single perspective. At the same time, the target detection algorithm uses the yolov5 detector for 2D detection of objects.

[0006] In "A Fully Automatic Parking Positioning and Mapping Method Based on Surround View Vision", the serious distortion problem of fisheye images is mainly solved by de-distortion. However, removing distortion means there will be a loss of perspective, which requires more fisheye cameras to overlap their perspectives to reduce the loss of perspective. Summary of the invention

[0007] The present invention provides a 3D perception method of a surround fisheye image based on spherical convolution and related equipment, which solves the problem that the input image of the existing fisheye camera does not take the surround image into consideration and more fisheye cameras are needed for overlapping viewing angles.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A 3D perception method for surround fisheye images based on spherical convolution, comprising:

[0010] S1 stitches multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image;

[0011] S2 inputs the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group;

[0012] S3 performs 1×1 convolution on the obtained feature map channel by channel to obtain a reduced-dimensional feature map on the surface point set of the unit sphere, and directly adds the reduced-dimensional feature maps of multiple convolutional layers through residual connection;

[0013] S4 performs feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling;

[0014] S5 performs steps S2 and S3 on the feature map after feature pooling to obtain a feature map with secondary dimensionality reduction, and adds the feature map to the feature map after feature pooling to obtain a fused feature map;

[0015] S6 repeatedly obtains multiple groups of fused feature maps of different sizes to obtain multi-scale features, and uses the multi-scale features in the 3D perception task head to obtain 3D perception content.

[0016] Preferably, the step of stitching a plurality of panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection is specifically as follows:

[0017] Align the optical centers of each camera, use their geometric centers as the virtual optical center, and construct a set of unit sphere surface points with the virtual optical center as the center. Use the respective camera models to project each fisheye image onto the spherical surface, and map the spherical image to the corresponding plane coordinates through latitude and longitude coordinates to generate a continuous and wide-angle equirectangular panorama.

[0018] Preferably, when generating a continuous and wide-angle equirectangular panorama, in order to obtain a coherent image, a weighted sum is performed in the image overlap area, and the specific method is as follows:

[0019]

[0020] In the formula, w i Represents pixels The weight of is the pixel to the center of the fisheye image, c i The Euclidean distance, r i is the maximum distance from the center to the edge.

[0021] Preferably, the process of projecting each fisheye image onto a spherical surface using the respective camera model is:

[0022]

[0023] φ=arctan2(y,x)

[0024] Where θ∈[0,π] represents latitude, φ∈[-π,π] represents longitude, x, y, z are the coordinates of the point in the camera coordinate system, and r is the distance between a point in the camera coordinate system and the center of the unit sphere, which is the geometric center of the optical center of the surround camera group.

[0025] Preferably, the steps of equivariant feature extraction are specifically as follows:

[0026] Firstly, the convolution kernel is constructed, and the spherical image is parameterized by spherical coordinates to obtain the parameterized spherical image coordinates. The continuous spherical signal-shaped convolution in the discretized spherical image coordinates is defined, and then the convolution is accelerated by fast Fourier transform to obtain the feature map defined on the three-dimensional special orthogonal group.

[0027] Preferably, the convolution of the continuous spherical signal is defined as:

[0028]

[0029] Among them, f and h are continuous ball signals, h is the convolution kernel, R is the rotation matrix of the three-dimensional special orthogonal group, It is an invariant integral metric on the set of points on the surface of the unit sphere;

[0030] Fast Fourier transform accelerated convolution is specifically:

[0031] Ft(H*I s )=Ft(H)*Ft(I s ) *

[0032]

[0033] Among them, * represents the convolution operation of the unit sphere surface point set, represents the outer product of the input signal spectrum, Ft(·) represents the Fourier transform of the unit sphere surface point set, and the superscript * represents the complex conjugate. In this way, the extracted feature F = H*I s ∈R C×H×W×G is the characteristic graph of the three-dimensional special orthogonal group.

[0034] Preferably, the step of performing channel-by-channel 1×1 convolution on the obtained feature map to obtain a reduced-dimensional feature map on a set of unit sphere surface points is specifically as follows:

[0035]

[0036] Where C represents the number of channels, G represents the number of rotational components, i, j, k and l represent the position of the channel, the vertical position on the panorama, the horizontal position on the panorama and the rotation axis on the panorama respectively.

[0037] A surround fisheye image 3D perception system based on spherical convolution, comprising:

[0038] Sampling module: used to stitch multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image;

[0039] Equivariant feature extraction module: used to input the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group;

[0040] Dimensionality reduction module: used to perform 1×1 convolution on the obtained feature map channel by channel to obtain the reduced dimension feature map on the surface point set of the unit sphere, and directly add the reduced dimension feature maps of multiple convolutional layers through residual connection;

[0041] Feature pooling module: used to perform feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling;

[0042] Feature map fusion module: used to process the feature map after feature pooling in steps S2 and S3 to obtain a feature map with secondary dimensionality reduction, and add it to the feature map after feature pooling to obtain a fused feature map;

[0043] Perception module: It is used to repeatedly obtain multiple sets of fused feature maps of different sizes to obtain multi-scale features. The multi-scale features are used in the 3D perception task head to obtain 3D perception content.

[0044] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a method for 3D perception of a surround fisheye image based on spherical convolution are implemented.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a 3D perception method for surround-view fisheye images based on spherical convolution.

[0046] Compared with the prior art, the present invention has the following beneficial effects: The present invention provides a 3D perception method for surround fisheye images based on spherical convolution. By processing surround fisheye images through panoramic stitching and spherical convolution, the loss of viewing angle is greatly reduced. We project multiple fisheye images into equirectangular panoramas to maximize the preservation of image edge features, which is a major pain point in fisheye vision tasks. On the other hand, the spherical convolution method enables the convolutional neural network to learn more information from the distortion features of the surround fisheye images, and input the panorama into the spherical convolution network we constructed for multi-layer feature extraction, thereby improving the accuracy of obstacle recognition at the edge of the fisheye image and the representation of the distant view occupancy grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a 3D perception method for surround fisheye images based on spherical convolution of the present invention;

[0048] Figure 2 This is a diagram showing the arrangement and weighted fusion effect of the surround-view fisheye camera according to an embodiment of the present invention;

[0049] Figure 3 This is a logic diagram of a 3D perception method for surround-view fisheye images based on spherical convolution according to an embodiment of the present invention;

[0050] Figure 4 This is a block diagram of a surround fisheye image 3D perception system based on spherical convolution in the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0054] like Figure 1 As shown, an embodiment of the present invention provides a 3D perception method of a surround fisheye image based on spherical convolution, comprising:

[0055] S1 stitches multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image;

[0056] S2 inputs the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group;

[0057] S3 performs 1×1 convolution on the obtained feature map channel by channel to obtain a reduced-dimensional feature map on the surface point set of the unit sphere, and directly adds the reduced-dimensional feature maps of multiple convolutional layers through residual connection;

[0058] S4 performs feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling;

[0059] S5 performs steps S2 and S3 on the feature map after feature pooling to obtain a feature map with secondary dimensionality reduction, and adds the feature map to the feature map after feature pooling to obtain a fused feature map;

[0060] S6 repeatedly obtains multiple groups of fused feature maps of different sizes to obtain multi-scale features, and uses the multi-scale features in the 3D perception task head to obtain 3D perception content.

[0061] For S1, specifically, first, the images from multiple fisheye cameras are stitched together using a seamless weighted fusion method to generate a continuous and seamless panorama. This processing step can effectively avoid stitching artifacts between images and ensure the visual consistency of the panorama. Then, the generated panorama is projected into a spherical coordinate system to form a spherical image. This spherical representation method can maintain the geometric integrity of the image while making full use of the wide viewing angle characteristics of the fisheye camera.

[0062] The specific method starts with aligning the optical centers of each camera. Assuming that the optical centers of these fisheye cameras are almost the same, use their geometric centers as the virtual optical center C and construct a unit sphere S with C as the center. 2The purpose of this step is to ensure that all images are processed in the same spherical coordinate system, thereby simplifying the subsequent image stitching and feature extraction process. f Each fisheye image is projected onto a spherical surface. Finally, the spherical image is mapped to the corresponding plane coordinates through the latitude and longitude coordinates to generate a continuous, wide-angle equirectangular panorama. Specifically, this step involves expanding each pixel in the spherical image into the plane coordinate system according to its spherical coordinates (latitude and longitude) to generate an equirectangular panorama. For a point (x, y, z) in the camera coordinate system, the equirectangular projection process can be expressed as:

[0063]

[0064] φ=arctan2(y,x)

[0065] Where θ∈[0,π] represents latitude and φ∈[-π,π] represents longitude. In actual processing, the calculation is performed by back-projecting the points in the camera coordinate system corresponding to each pixel of the panorama into the fisheye image, that is, rendering is performed using ray tracing. s =(x s ,y s ,z s ) is a point in the camera coordinate system corresponding to a pixel of the panoramic image. The longitude and latitude of the pixel are known to be (φ,θ), and the imaging model of the fisheye camera is P(·):

[0066] x s = sinθ·cosφ

[0067] y s = sinθ·sinφ

[0068] z s = cosθ

[0069] I f =P(x s )

[0070] Since the fields of view of different cameras overlap, a pixel u on the panorama s May correspond to pixels u on multiple fisheye images f To obtain a coherent image, a weighted sum is performed in the overlapping region. s Corresponding pixel set Its value is calculated by weighted summation:

[0071]

[0072] For the calculation of this weight, this paper proposes a method different from the traditional method. Figure 3As shown in the figure, the traditional method assumes that the layout of the surround view cameras is horizontal and the overlapping area is a simple rectangle; however, there are complex camera layouts, such as the regular tetrahedral (methane) layout (Tetrahedral), whose overlapping area is complex and cannot be calculated by the traditional method. The specific method is as follows:

[0073]

[0074] Here i Represents pixels The weight of is the pixel to the center of the fisheye image, c i The Euclidean distance, r i is the maximum distance from the center to the edge. This method significantly improves the visual consistency and clarity of overlapping areas by giving priority to center pixels. Figure 3 As shown in the figure, the traditional method cannot handle the stitching and beautification of the regular tetrahedron camera configuration, while the method proposed in this paper and the simple weighted average comparison can significantly eliminate ghosting and optimize the quality of the panorama.

[0075] For S2, after converting the surround fisheye images into equirectangular panoramas, these panoramas are processed using spherical convolutional neural networks (SCNNs) to extract rotational equivariant features. The process of equivariant feature extraction includes the following steps: First, the convolution kernel is constructed. Equirectangular panorama I s ∈R C×H×W Can be regarded as a unit ball S 2 The discrete spherical signal on has C channels. S is parameterized by the spherical (longitude and latitude) coordinates (φ,θ) 2 , where W and H represent the discretized φ and θ respectively. In the equivariant feature extraction process, the convolution kernel H∈R C×h×w , defined as S 2 The discrete spherical signal on is located near the north pole of the unit sphere. Before introducing the discrete spherical signal convolution, the continuous spherical signal convolution is defined as follows:

[0076]

[0077] Where f and h are both continuous ball signals, h is the convolution kernel, R is the rotation matrix in SO(3), YesS 2 The convolution process ensures the consistency of feature extraction results under different rotations. Since the volume enclosed by the ball signal remains unchanged when it rotates around its center, it can be deduced that:

[0078]

[0079] This formula marks S2 Equivariance of convolution. This formula can be extended to discrete spherical signal convolution

[12] :

[0080] L Q (H*I s )=H*(L Q I s )

[0081] In order to achieve efficient equivariant feature extraction, S 2 Convolution theorem, and accelerate the convolution process through fast Fourier transform (FFT):

[0082] Ft(H*I s )=Ft(H)*Ft(I s ) *

[0083]

[0084] Among them, * represents S 2 Convolution operation, represents the outer product of the input signal spectrum, Ft(·) represents S 2 Fourier transform, superscript * indicates complex conjugate. In this way, the extracted feature F = H * I s ∈R C×H×W×G is the SO(3) signal with rotational equivariance. If the SO(3) space is parameterized by ZYZ Euler angles (φ,θ,γ)∈[-π,π]×[0,π]×[-π,π], H, W and G represent the discretization accuracy of φ, θ and γ respectively.

[0085] For S3, the extracted features F∈R C×H×W×G is a SO(3) signal. To construct an equivariant deep convolutional neural network, an additional dimension G must be considered. The existing SO(3) signal convolution method uses FFT calculation with a time complexity of O(bw 4 ), and the three-dimensional SO(3) features require a lot of storage. In order to reduce the computational and storage burden, this paper proposes a dimensionality reduction method that maintains rotational equivariance and integrates the information on the rotation dimension through weighted summation:

[0086]

[0087] The pooling operation is required to integrate the features of the SO(3) signal into S 2 Signal Keep the rotation equivariance of convolution. Through this method, a deeper SCNN structure can be built without increasing complexity, optimizing computing resources and storage requirements. The specific formula is as follows:

[0088]

[0089] The dimension of the weight matrix c is R C×G , where C represents the number of channels and G represents the number of rotational components. These weight matrices c are learnable parameters that can be optimized during the training process. The indices i, j, k, and l represent the channel, the vertical position on the panorama, the horizontal position, and the rotation axis, respectively. With this design, the features of each channel can adapt to different rotational components, thereby achieving adaptive rotational feature integration of the input signal. This learnable pooling strategy effectively reduces the dimension while retaining key rotational features, thereby improving the efficiency and performance of the convolutional network. Specifically, by introducing a weighted summation method in the pooling operation, rotational equivariance is ensured. Rotational equivariance refers to the property that features remain unchanged under rotational transformations, which is particularly important for processing wide-angle views in fisheye images. Using learnable weights, the pooling strategy can adaptively integrate the information of different rotational components into each channel, thereby maintaining the transformation characteristics of the original signal. At the same time, the S of multiple convolutional layers are connected by residual connections. 2 Features are summed directly, which ensures that information propagates quickly in the network while alleviating the vanishing gradient problem.

[0090] For S4 and S5, specifically, S3 obtained by 2 The feature map is subjected to maximum pooling (neighborhood maximum sampling) to obtain a small-sized feature map. The spherical map is downsampled, and then added to the above-mentioned pooled small-sized feature map through S2 and S3 to obtain a feature map of two sizes. Repeat the above operation to change the downsampling size to obtain a multi-size feature pyramid, generally four sizes are sufficient.

[0091] Finally, the multi-scale features obtained by S5 are connected to the 3D perception task head, such as the 3D semantic occupancy grid prediction task head and the 3D object detection task head to obtain the final algorithm output. Figure 3 shown.

[0092] Specifically, 3D semantic occupancy grid prediction is an emerging task in the field of computer vision and artificial intelligence, which aims to predict the occupancy state and semantic category of each voxel in three-dimensional space. This task requires the model to not only recognize the existence of objects, but also accurately distinguish the types and shapes of objects. Specifically, 3D semantic occupancy grid prediction converts input sensor data (such as LiDAR point clouds and camera images) into a three-dimensional grid map containing semantic labels, thereby providing detailed semantic information of the environment. The 3D semantic occupancy grid prediction in this paper is based on SurroundOcc. This method significantly improves the accuracy of occupancy grid prediction by fusing multi-view image information (pure vision).

[0093] In the SurroundOcc model, the input multi-view image is firstly extracted through a convolutional neural network (CNN) to extract feature representation. i , CNN extracts the feature map F i :

[0094] F i =CNN(I i )

[0095] These feature maps are then projected into three-dimensional space through a projection function P to generate voxel features V i In the feature fusion of 2D image and 3D voxel features, a multi-scale deformable cross attention mechanism (Deformable Attention, DeformAttn) is adopted:

[0096]

[0097] Where V p The pth element representing the voxel feature, Q p represents the query generated from the voxel features, q p represents the three-dimensional coordinates of the voxel feature in space, i represents the camera angle, ν hit Represents the pixel P(q p ,i) The set of subscripts of the image perspectives that are “hit”. Finally, the multi-scale voxel features V after 3D-FPN fusion are processed by a three-dimensional convolutional network to output the occupancy probability and semantic label of each voxel. This process can be expressed as:

[0098] O=3DCNN(V)

[0099] Among them, O∈H×W×Z×C is the output occupancy grid map, each voxel contains occupancy probability and semantic label. H×W×Z is the 3D size of the voxel perception range, which also represents the number of queries in the attention mechanism, and C is the semantic information. This paper adopts this task head to perform the occupancy grid prediction task. Through the spherical feature map I s It uses a spherical convolutional neural network to extract 2D features, uses the cross-attention of the fisheye camera model to increase the dimension of the features, and finally uses the prediction head to make predictions.

[0100] 3D Semantic Object Detection is an important task in the field of computer vision and artificial intelligence, which aims to detect and identify the position and category of objects in three-dimensional space. Compared with traditional two-dimensional object detection, 3D object detection needs to process more complex spatial information and geometric relationships, and is often used in fields such as autonomous driving, robot navigation and augmented reality. In autonomous driving, accurate 3D object detection can help vehicles identify and locate pedestrians, vehicles and obstacles around them in real time, thereby achieving safe and efficient driving decisions; in robot navigation, 3D object detection can provide robots with detailed environmental perception, ensuring accurate obstacle avoidance and autonomous movement in complex environments; in augmented reality, 3D object detection can enhance the interactive experience between the virtual and real worlds, providing a more realistic and immersive user experience.

[0101] The 3D object detection method in this paper is based on the DETR3D (Detection Transformer for 3D Object Detection) model. The DETR3D model is an innovative 3D object detection method that implements end-to-end 3D object detection by introducing the Transformer structure. Unlike traditional convolutional neural networks (CNNs), the Transformer model captures global features through a self-attention mechanism, thereby improving the model's ability to understand complex spatial information and geometric relationships.

[0102] In order to adapt to the wide-angle view characteristics of fisheye images, this paper converts the traditional planar CNN into a spherical CNN and adopts a cross-attention mechanism based on the camera model. The spherical CNN can better adapt to the wide-angle characteristics of fisheye images and capture more environmental information by performing convolution operations on the spherical surface. The cross-attention mechanism further improves the accuracy and consistency of feature extraction by fusing features between different perspectives.

[0103] In the specific implementation process, the input fisheye image is first extracted through the spherical CNN feature representation. The extracted feature maps are fused through the cross-attention mechanism to generate the final 3D object detection feature set O∈R^(N×C), where N is the number of cross-attention queries and C is the target prediction information, including position, scale, rotation angle, and category. Specifically, C is in the form of [x, y, z, dx, dy, dz, yaw, class], where (x, y, z) represents the position of the 3D bounding box, (dx, dy, dz) represents its scale, yaw represents the rotation angle of the bounding box around the Z axis of its own coordinate system, and class represents the predicted label of the target.

[0104] In this setting, the process of 3D object detection can be described in the following steps:

[0105] 1. Feature extraction: First, features are extracted from the input fisheye image through a spherical CNN. The spherical CNN can perform convolution operations on the spherical surface to capture the wide-angle information in the fisheye image, thereby generating high-quality feature maps.

[0106] 2. Feature fusion: Next, the extracted feature maps are fused through the cross-attention mechanism. The cross-attention mechanism ensures the accuracy and consistency of feature extraction by fusing features between different perspectives, thereby generating a more accurate 3D feature representation.

[0107] 3. Object detection: Finally, the fused feature map is input into the 3D object detection module to generate a 3D object detection feature set O. Through this process, the model can predict the position, scale, rotation angle and category of each 3D object, and achieve accurate detection of objects in three-dimensional space.

[0108] like Figure 4 As shown, the embodiment of the present invention further provides a surround fisheye image 3D perception system based on spherical convolution, comprising:

[0109] Sampling module: used to stitch multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image;

[0110] Equivariant feature extraction module: used to input the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group;

[0111] Dimensionality reduction module: used to perform 1×1 convolution on the obtained feature map channel by channel to obtain the reduced dimension feature map on the surface point set of the unit sphere, and directly add the reduced dimension feature maps of multiple convolutional layers through residual connection;

[0112] Feature pooling module: used to perform feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling;

[0113] Feature map fusion module: used to process the feature map after feature pooling in steps S2 and S3 to obtain a feature map with secondary dimensionality reduction, and add it to the feature map after feature pooling to obtain a fused feature map;

[0114] Perception module: It is used to repeatedly obtain multiple sets of fused feature maps of different sizes to obtain multi-scale features. The multi-scale features are used in the 3D perception task head to obtain 3D perception content.

[0115] A terminal device is provided in one embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0116] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0117] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0118] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0119] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0120] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0121] Although the embodiments of the present invention are described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments and application fields, and the above-mentioned specific embodiments are only illustrative and instructive, rather than restrictive. Under the guidance of the specification, a person skilled in the art can also make many forms without departing from the scope of protection of the claims of the present invention, all of which belong to the scope of protection of the present invention.

Claims

1. A 3D perception method for surround fisheye images based on spherical convolution, characterized in that: include: S1 stitches multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image; S2 inputs the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group; S3 performs 1×1 convolution on the obtained feature map channel by channel to obtain a reduced-dimensional feature map on the surface point set of the unit sphere, and directly adds the reduced-dimensional feature maps of multiple convolutional layers through residual connection; S4 performs feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling; S5 performs steps S2 and S3 on the feature map after feature pooling to obtain a feature map with secondary dimensionality reduction, and adds the feature map to the feature map after feature pooling to obtain a fused feature map; S6 repeatedly obtains multiple groups of fused feature maps of different sizes to obtain multi-scale features, and uses the multi-scale features in the 3D perception task head to obtain 3D perception content.

2. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 1, characterized in that: The specific steps of stitching multiple panoramic fisheye images sampled at the same time into an equirectangular projection panorama are: Align the optical centers of each camera, use their geometric centers as the virtual optical center, and construct a set of unit sphere surface points with the virtual optical center as the center. Use the respective camera models to project each fisheye image onto the spherical surface, and map the spherical image to the corresponding plane coordinates through latitude and longitude coordinates to generate a continuous and wide-angle equirectangular panorama.

3. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 2, characterized in that: When generating a continuous and wide-angle equirectangular panorama, in order to obtain a coherent image, a weighted sum is performed in the overlapping areas of the images as follows: In the formula, w i Represents pixels The weight of is the pixel to the center of the fisheye image, c i The Euclidean distance, r i is the maximum distance from the center to the edge.

4. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 2, characterized in that: The process of projecting each fisheye image onto a spherical surface using its respective camera model is: φ=arctan2(y,x) Where θ∈[0,π] represents latitude, φ∈[-π,π] represents longitude, x, y, z are the coordinates of the point in the camera coordinate system, and r is the distance between a point in the camera coordinate system and the center of the unit sphere, which is the geometric center of the optical center of the surround camera group.

5. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 1, characterized in that: The specific steps of equivariant feature extraction are: Firstly, the convolution kernel is constructed, and the spherical image is parameterized by spherical coordinates to obtain the parameterized spherical image coordinates. The continuous spherical signal-shaped convolution in the discretized spherical image coordinates is defined, and then the convolution is accelerated by fast Fourier transform to obtain the feature map defined on the three-dimensional special orthogonal group.

6. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 1, characterized in that: The convolution of continuous spherical signals is defined as: Among them, f and h are continuous ball signals, h is the convolution kernel, R is the rotation matrix of the three-dimensional special orthogonal group, It is an invariant integral metric on the set of points on the surface of the unit sphere; Fast Fourier transform accelerated convolution is specifically: Ft(H*I s )=Ft(H)*Ft(I s ) * Among them, * represents the convolution operation of the unit sphere surface point set, represents the outer product of the input signal spectrum, Ft(·) represents the Fourier transform of the unit sphere surface point set, and the superscript * represents the complex conjugate. In this way, the extracted feature F = H*I s ∈R C ×H×W×G is the characteristic graph of the three-dimensional special orthogonal group.

7. The method for 3D perception of surround fisheye images based on spherical convolution according to claim 1, characterized in that: The steps of performing 1×1 convolution on the obtained feature map channel by channel to obtain the reduced dimension feature map on the surface point set of the unit sphere are as follows: Where C represents the number of channels, G represents the number of rotational components, i, j, k and l represent the position of the channel, the vertical position on the panorama, the horizontal position on the panorama and the rotation axis on the panorama respectively.

8. A surround fisheye image 3D perception system based on spherical convolution, characterized in that: include: Sampling module: used to stitch multiple panoramic fisheye images sampled at the same time into a panoramic image of equirectangular projection, i.e., a spherical image; Equivariant feature extraction module: used to input the spherical image into the spherical convolutional neural network to perform equivariant feature extraction and obtain a feature image defined on a three-dimensional special orthogonal group; Dimensionality reduction module: used to perform 1×1 convolution on the obtained feature map channel by channel to obtain the reduced dimension feature map on the unit sphere surface point set, and directly add the reduced dimension feature maps of multiple convolutional layers through residual connection; Feature pooling module: used to perform feature pooling on the added reduced-dimensional feature map to obtain a feature map after feature pooling; Feature map fusion module: used to process the feature map after feature pooling in steps S2 and S3 to obtain a feature map with secondary dimensionality reduction, and add it to the feature map after feature pooling to obtain a fused feature map; Perception module: It is used to repeatedly obtain multiple sets of fused feature maps of different sizes to obtain multi-scale features. The multi-scale features are used in the 3D perception task head to obtain 3D perception content.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for 3D perception of surround fisheye images based on spherical convolution as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of a method for 3D perception of surround fisheye images based on spherical convolution as claimed in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Image processing method, system and device based on spherical convolutional neural network

    CN119991421A