Vehicle detection method and system based on bionic compound eye array camera

Abstract

The invention discloses a vehicle detection method based on a bionic compound eye array camera, and belongs to the technical field of automatic driving. Comprising the steps of obtaining multi-view single-camera images, performing feature extraction on the multiple single-camera images, and generating corresponding camera features and semantic segmentation prediction results; processing camera features and semantic segmentation prediction results through an attention mechanism based on semantic information, and mapping the camera features and the semantic segmentation prediction results to a three-dimensional grid map to obtain voxel features; the voxel features are processed through an attention mechanism based on depth information, and final feature representation is generated; and fusing the multi-view final feature representation by combining 3D convolution and deformable convolution, generating an occupation condition of a three-dimensional grid, and judging whether a vehicle exists in the three-dimensional grid. The multi-view information of the camera overlapping area can be fused, and the problem that the detection accuracy of vehicle perception and detection under the large view field angle needs to be improved is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a vehicle detection method and system based on a bionic compound eye array camera. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Research on biomimetic compound eye array cameras began with in-depth observation and imitation of insect visual systems, particularly the superior performance of insect compound eyes in terms of large field of view, rapid response, and multi-target tracking. Insect compound eyes are composed of a large number of simple photosensitive units (called ommatidia), each with a fixed viewing angle, which together cover a portion of the field of view. By integrating this information, insects can rapidly perceive the motion of objects over a wide range. Based on these characteristics, the compound eye structure has inspired many researchers to study biomimetic visual systems.

[0004] Early research on biomimetic compound eye cameras focused on simulating the wide field of view characteristics of insect vision. With the development of mathematical models for compound eye imaging, the geometric properties and optical perception principles of insect compound eyes gradually became known. Subsequently, scientists leveraged this model to construct biomimetic camera systems with wide field of view capabilities. Simultaneously, advances in microelectronics manufacturing technologies enabled the creation of biomimetic compound eye camera systems with wide field of view and high sensitivity. By integrating multiple low-resolution sensors into a single device, the basic model of a biomimetic compound eye array camera was established.

[0005] With the widespread adoption of biomimetic compound eye cameras, effectively fusing the overlapping fields of view of multiple low-resolution cameras has become a key research topic. Most current research focuses on aligning and fusing overlapping fields of view captured by different cameras to achieve a broader scene perception. Specifically, by stitching images from array cameras together, a single image with a wide field of view is generated, enabling tasks such as object detection in large field-of-view environments.

[0006] Vehicle detection is a key task in applications such as autonomous driving, intelligent transportation, and urban monitoring. With the advancement of computer vision, vehicle detection technology has evolved from object detection in 2D images to today's 3D object detection. 3D object detection not only provides more accuracy in the spatial positioning of vehicles, but also lays a foundation for predicting object behavior, route planning, and environmental understanding. Occupancy Network (Occ) is a new technology for 3D object detection proposed in recent years. Its working principle is to map the input data from multi-view images into a three-dimensional feature space, and then learn the occupancy probability of each point in the feature space through a deep neural network.

[0007] However, existing Occ requires high computing resources and is difficult to effectively utilize overlapping information between cameras under conventional camera layout specifications. Summary of the Invention

[0008] In order to address the shortcomings of the existing technology, the present invention provides a bionic compound eye array camera system, a vehicle detection method and system based on the bionic compound eye array camera, an electronic device, a computer-readable storage medium and a computer program product. Through the innovative application of the bionic compound eye array camera, combined with the deep fusion of multi-perspective information, and through the introduction of semantic and deep attention mechanisms, vehicle perception and detection under a large field of view are achieved, thereby improving the accuracy of vehicle detection.

[0009] In a first aspect, the present invention provides a bionic compound eye array camera system;

[0010] A bionic compound eye array camera system, comprising a multi-layer monocular camera and a spherical bracket;

[0011] Multiple layers of single-lens cameras are arranged on the spherical bracket along a longitudinal spacing, and adjacent layers of single-lens cameras are arranged in parallel and staggered manner;

[0012] Each layer of monocular cameras includes a plurality of monocular cameras, and the plurality of monocular cameras are arranged on the spherical bracket along a lateral spacing.

[0013] In a second aspect, the present invention provides a vehicle detection method based on a bionic compound eye array camera;

[0014] A vehicle detection method based on a bionic compound eye array camera includes:

[0015] Acquire multi-view single-camera images, perform feature extraction on each of the multiple single-camera images, and generate corresponding camera features and semantic segmentation prediction results; wherein the multi-view single-camera images are collected by a bionic compound eye array camera system;

[0016] The camera features and semantic segmentation prediction results are processed and mapped to a 3D grid map through an attention mechanism based on semantic information to obtain voxel features. The voxel features are processed through an attention mechanism based on depth information to generate the final feature representation.

[0017] The final feature representations of multiple views are fused by combining 3D convolution and deformable convolution to generate the occupancy of the 3D grid and determine whether there is a vehicle in the 3D grid.

[0018] In some embodiments, further comprising:

[0019] Get a three-dimensional occupancy grid sequence;

[0020] The three-dimensional occupancy grid sequence is processed through the trained time series feature extraction network to obtain the occupancy probability after time series fusion and determine whether there is a vehicle.

[0021] In some embodiments, extracting features from multiple single-camera images to generate corresponding camera features and semantic segmentation prediction results specifically includes:

[0022] The Conv-UNext network is used to extract features from multiple single-camera images in sequence to obtain the corresponding camera features. The camera features are processed in sequence by depth-wise separable convolution and Sigmoid activation function to generate semantic segmentation prediction results.

[0023] In some embodiments, processing the camera features and the semantic segmentation prediction results by using an attention mechanism based on semantic information and mapping them to a three-dimensional grid map to obtain voxel features specifically includes:

[0024] Multiply the camera features by the corresponding semantic segmentation prediction results element by element to obtain the intermediate feature representation;

[0025] Calculate the foreground ratio of each monocular camera and determine the attention weight of each monocular camera based on the foreground ratio; obtain weighted features based on the attention weight and intermediate feature representation;

[0026] According to the coordinates and weighted features of each voxel in the three-dimensional grid map on the image plane, the corresponding voxel features are obtained.

[0027] In some embodiments, processing voxel features using an attention mechanism based on depth information to generate a final feature representation specifically includes:

[0028] Calculate the baseline depth of the 3D grid map and the estimated depth value of each single camera, and determine the attention weight that combines the depth information;

[0029] Based on the attention weights and voxel features, the final feature representation is generated.

[0030] In a third aspect, the present invention provides a vehicle detection system based on a bionic compound eye array camera;

[0031] The vehicle detection system based on the bionic compound eye array camera includes:

[0032] A feature extraction module is configured to: obtain single-camera images from multiple perspectives, perform feature extraction on each of the multiple single-camera images, and generate corresponding camera features and semantic segmentation prediction results;

[0033] The feature generation module is configured to: process the semantic segmentation prediction results through an attention mechanism based on semantic information and map them to a three-dimensional grid map to obtain voxel features; process the voxel features through an attention mechanism based on depth information to generate a final feature representation;

[0034] The multi-camera feature fusion module is configured to fuse the final feature representations of multiple perspectives by combining 3D convolution and deformable convolution, generate the occupancy of the 3D grid, and determine whether there is a vehicle in the 3D grid.

[0035] In a fourth aspect, the present invention provides an electronic device;

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-mentioned vehicle detection method based on a bionic compound eye array camera.

[0037] In a fifth aspect, the present invention provides a computer-readable storage medium;

[0038] A computer-readable storage medium stores a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned vehicle detection method based on a bionic compound eye array camera.

[0039] In a sixth aspect, the present invention provides a computer program product;

[0040] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned vehicle detection method based on a bionic compound eye array camera.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] 1. The technical solution provided by the present invention achieves efficient perception with a large field of view through a bionic compound eye array camera system. It can effectively simulate the wide field of view advantage of insect compound eyes, significantly improving the detection system's ability to capture multi-view information over a large range, making vehicle detection coverage more extensive and able to meet the needs of panoramic perception in complex environments.

[0043] 2. The technical solution provided by the present invention introduces a dual attention mechanism based on semantic information and depth information to achieve precise perception of the multi-level attention mechanism, enabling the detection method to focus on the key vehicle area and effectively suppress background interference; in feature processing, the semantic information is first used to weight the vehicle area, and then the depth information is used to further accurately determine whether a vehicle exists, achieving high-precision perception in three-dimensional space.

[0044] 3. The technical solution provided by this invention improves multi-camera information fusion. By combining 3D convolution with deformable 3D convolution, it enables better alignment and fusion of features in overlapping areas across cameras, significantly enhancing the comprehensive expressive power of multi-view features. Furthermore, through residual connections and feature dimensionality reduction strategies, it effectively reduces the computational burden while maintaining the reliability of the multi-camera fusion results, ensuring the accuracy of occupancy information in three-dimensional space.

[0045] 4. The technical solution provided by the present invention enhances the stability of vehicle detection through temporal information fusion, can smooth the motion changes of objects between consecutive frames, reduce detection instability caused by instantaneous occlusion or perspective changes, ensure stable and accurate tracking and detection of vehicles in dynamic scenes, and provide higher reliability for complex scene perception in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0047] Figure 1 A schematic flow chart of a vehicle detection method provided by an embodiment of the present invention;

[0048] Figure 2 A schematic diagram of the network architecture of the Conv-UNext encoder provided in an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the data collection process in a simulation experiment provided by an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the process of data preprocessing in a simulation experiment provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0053] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0054] Example 1

[0055] The bionic compound eye array design achieves full field of view coverage, capable of capturing environmental information across a wide area around the vehicle, making it particularly suitable for detecting complex targets in dynamic scenes. Compared to traditional monocular cameras or non-overlapping multi-cameras, the compound eye array significantly improves its ability to capture environmental details.

[0056] This embodiment provides a biomimetic compound eye array camera system that captures information from multiple perspectives by simulating the multi-eye vision characteristics of insect compound eyes. This system aims to provide wider field of view coverage and more detailed capture of the environment, offering significant advantages in vehicle detection tasks.

[0057] Insect compound eyes are composed of multiple, independently functional monoculars, each with a limited field of view. Through the coordinated arrangement of these monoculars, insect compound eyes can achieve wide-area visual coverage and rapidly perceive targets in dynamic environments. This embodiment provides a biomimetic compound eye array camera system comprising multiple layers of monocular cameras and a spherical support. The multiple layers of monocular cameras are mounted on the spherical support at longitudinal intervals along the sphere, with adjacent layers of monocular cameras arranged in parallel and staggered configurations. Each layer of monocular cameras includes multiple monocular cameras, which are mounted on the spherical support at transverse intervals. This multi-layer, parallel, staggered arrangement achieves optimized field of view coverage.

[0058] In one embodiment, a bionic compound eye array camera system for providing multi-view information includes three layers of monocular cameras arranged in a parallel, staggered pattern. From top to bottom, the number of monocular cameras in each layer is 6, 5, and 6, respectively. Specifically, each monocular camera is a low-resolution RGB camera with a resolution of 120×120 pixels and a field of view of 60°. The monocular cameras are arranged on a spherical bracket with a diameter of 20 cm. The angle between the monocular cameras in each layer is 30°, and the angle between layers is 15°.

[0059] Under these conditions, the system can achieve sensing within a 50-meter range in front of, to the left and right of the vehicle, with at least three monocular cameras covering any location within this range. Furthermore, this arrangement allows for easy adjustment of the number of cameras to achieve different sensing ranges.

[0060] This arrangement can not only capture important information in the environment, but also reduce data redundancy and computing overhead. The compound eye array system can provide sufficient overlapping areas while maintaining a wide viewing angle. Information fusion in overlapping areas can effectively improve the accuracy of vehicle detection, especially in detection tasks in complex scenes around the vehicle. This embodiment achieves efficient and comprehensive information collection through a bionic compound eye array camera system, laying a solid foundation for subsequent multi-camera information fusion, feature extraction, and dynamic detection in complex scenes. This design fully utilizes the advantages of bionics, combining the wisdom of nature with artificial vision systems, and provides innovative solutions for fields such as autonomous driving.

[0061] Example 2

[0062] Existing vehicle detection methods lack the utilization of overlapping information between cameras, resulting in the need to improve the accuracy and efficiency of vehicle detection in dynamic scenes. Therefore, the present invention provides a vehicle detection method based on a bionic compound eye array camera, focusing on vehicle detection based on a bionic compound eye array camera. By fusing multi-view information in the overlapping areas of the cameras, vehicle perception and detection in a large field of view are achieved.

[0063] Next, combine Figures 1-4 , a vehicle detection method based on a bionic compound eye array camera disclosed in this embodiment is described in detail. The vehicle detection method based on a bionic compound eye array camera includes the following steps:

[0064] S1. Obtain single-camera images from multiple perspectives, perform feature extraction on multiple single-camera images, and generate corresponding camera features and semantic segmentation prediction results.

[0065] As an implementation method, S1 specifically includes:

[0066] S101. Perform feature extraction on multiple single-camera images in sequence through the Conv-UNext network to obtain corresponding camera features.

[0067] In this embodiment, the Conv-UNext network includes a downsampling layer, four downconvolution blocks, four upconvolution blocks and an upconvolution layer connected in sequence, and the downconvolution block is residually connected to the upconvolution block; that is, the output of the third downconvolution block is fused with the output of the first upconvolution block and input into the second upconvolution block, the output of the second downconvolution block is fused with the output of the second upconvolution block and input into the third upconvolution block, and the output of each downconvolution block is fused with the output of the third upconvolution block and input into the fourth upconvolution block.

[0068] The lower convolution block includes a 7×7 convolution layer, a normalization layer, a 1×1 convolution layer, a GELU, and a 1×1 convolution layer connected in sequence. The output of the 7×7 convolution layer is residually connected to the output of the last 1×1 convolution layer. The upper convolution block includes a 7×7 transposed convolution layer, a normalization layer, a 1×1 convolution layer, a GELU, and a 1×1 convolution layer connected in sequence. The output of the 7×7 transposed convolution layer is residually connected to the output of the last 1×1 convolution layer.

[0069] The Conv-UNext network includes designs such as 7×7 window convolution kernels, depthwise separable convolutions, and an inverse bottleneck structure. While effectively maintaining the local receptive field advantage of the convolutional network, it significantly reduces the amount of computation through large convolution kernels and simplified activation function / normalization layer settings, thereby enhancing the network's training stability and robustness.

[0070] Specifically, in the encoder of the Conv-UNext network, first, the input single-camera image is downsampled and the image size is adjusted through Layer Normalization (LN) and convolutional layers.

[0071] For example, if the size of the input single camera image is H×W and the number of feature channels is C, then the size of the feature map after downsampling can be expressed as:

[0072]

[0073] Where X is a single camera image, Conv(·) represents the convolution operation, and H and W are the height and width of the single camera image, respectively.

[0074] In each subsequent block, in order to reduce the amount of computation, this embodiment reduces the dimensionality increase ratio of the inverse bottleneck structure from the original 4 times to 2 times. The depthwise separable convolution is used to further reduce the number of model parameters while still maintaining a large convolution kernel size (7×7) to expand the receptive field and improve the efficiency of feature extraction. The output of the depthwise separable convolution in the block can be expressed as:

[0075]

[0076]

[0077] Among them, DepthwiseConv(·) represents depth-wise separable convolution, and PointwiseConv(·) represents point-by-point convolution.

[0078] In the decoder part, in this embodiment, in order to maintain the symmetry with the encoder structure and enhance the ability of feature upsampling, a 7×7 transposed convolution kernel is used, which can reduce information loss while enlarging the size of the feature map.

[0079] Assuming the size of the input feature map is H / 4×W / 4, after transposed convolution, it is restored to its original size:

[0080]

[0081] At this point, the camera feature F extraction of the input single camera image is completed.

[0082] S102: Process the camera features sequentially through depth-wise separable convolution and sigmoid activation function to generate semantic segmentation prediction results.

[0083] To further improve the model's detection performance, this example further processes the semantic and depth information in the image based on feature extraction. By using large convolution kernels and depthwise separable convolution to reduce the feature dimensionality, and combining it with a sigmoid function for normalization, the final semantic and depth segmentation predictions are obtained.

[0084] For example, let the camera feature be The output feature map after depth-separable convolution dimensionality reduction is:

[0085]

[0086] Next, two detection heads are used to process the output feature maps respectively to extract semantic information and depth information, that is, to generate a specific semantic feature map S through the convolution layer. m_a and deep feature map S ^_jnb :

[0087]

[0088] Among them, SegConv and DepthConv represent the semantic detection head and depth detection head respectively.

[0089] Then, the Sigmoid activation function is used to obtain the semantic segmentation prediction result P m_a And the depth segmentation prediction result P ^_jnb :

[0090] P m_a =σ(Sm_a );

[0091] P ^_jnb =σ(S ^_jnb ).

[0092] In this step, during the training process, the above model is trained using the semantic segmentation loss function and the depth estimation loss function.

[0093] For semantic segmentation, this embodiment uses Dice Loss as the loss function to measure the matching degree between the model prediction and the true label. Dice Loss is expressed as:

[0094]

[0095] Among them, p c ,g c They represent the predicted value and true label of the i-th pixel, respectively, and N is the total number of pixels.

[0096] By using Dice Loss, this paper can better optimize the semantic segmentation task in real time, especially when the target area is small. Dice Loss can effectively avoid the category imbalance problem and ensure more accurate segmentation results.

[0097] In order to improve the accuracy of depth estimation, this embodiment adopts a multi-camera joint depth estimation method. By fusing depth information from multiple camera perspectives, the consistency of depth estimation is optimized and the error is reduced. Suppose N cameras are synchronously shooting the same scene. For a certain real 3D coordinate point P in the scene, each camera calculates the depth estimation value of the point. The loss of depth estimation consists of two parts: absolute error weighted loss and consistency loss:

[0098] The absolute error loss directly compares the depth estimate d of each camera. c (P) and true depth d GT (P) difference, ensuring that the depth estimated by each camera gradually approaches the true value. It is defined as follows:

[0099]

[0100] Among them, w c is the confidence weight of camera i, d c (P) is P ^_jnb The depth estimation value of camera i to point P.

[0101] The consistency loss constrains the depth estimation results of multiple cameras to be as consistent as possible at the same real coordinate point P, reducing the estimation differences between cameras. For the depth estimation value d of two different cameras i and j c (P) and d d(P), the consistency loss is defined as:

[0102]

[0103] This loss measures the difference in depth estimation between different cameras, ensuring that the estimation results under multiple viewpoints are consistent, thereby improving the overall accuracy of depth estimation.

[0104] The final total depth estimation loss function combines the depth estimation error and consistency error under the confidence weight:

[0105]

[0106] Among them, λ1 and λ2 are the weight coefficients of the two loss terms, which are used to balance the depth estimation error and consistency optimization.

[0107] To ensure the effectiveness of optimization, the weight w c The confidence level of the depth estimation of each camera under a specific viewing angle is obtained by analyzing the depth estimation confidence level of each camera under a specific viewing angle. The confidence level can be calculated by combining the depth consistency with the surrounding cameras and the internal estimation accuracy of the current camera. The depth estimation value D of the real coordinate point P after optimization is P It can be expressed as:

[0108]

[0109] S2. The camera features and semantic segmentation prediction results are processed and mapped to a three-dimensional grid map through an attention mechanism based on semantic information to obtain voxel features.

[0110] Before mapping features to a 3D grid map, this embodiment introduces a semantic-based attention mechanism to enhance the expressiveness of features. This mechanism mainly includes two aspects: first, attention processing based on semantic results, and second, attention processing for each monocular camera.

[0111] As an implementation method, the specific process of processing camera features and semantic segmentation prediction results through the attention mechanism based on semantic information is as follows:

[0112] First, semantic-based pixel-level attention is applied to each camera feature, and the camera feature is element-wise multiplied with the corresponding semantic segmentation prediction result to obtain an intermediate feature representation.

[0113] For example, let the camera feature be F ][g , we can obtain the feature F after pixel-level attention processing ′ ][g , expressed as:

[0114] F ′ ][g =Pm_a ⊙F ][g ;

[0115] Where ⊙ represents element-wise multiplication.

[0116] Through this formula, it can be ensured that the characteristics of the vehicle area occupy a larger proportion in subsequent processing.

[0117] Then, for each monocular camera, calculate its foreground proportion P ` , defined as the ratio of vehicle pixels to total pixels in the camera. The foreground ratio is calculated as:

[0118]

[0119] Where N is the total number of pixels in a single monocular camera image.

[0120] In order to enhance the foreground information, the camera’s attention weight W is defined as k The relative weight of its prospect proportion is expressed as:

[0121]

[0122] Where M is the total number of cameras. In this way, cameras with a higher foreground ratio will receive a larger attention weight, allowing the attention mechanism to effectively focus on cameras containing more vehicle information.

[0123] Combining the above formulas, we can obtain the weighted feature F by semantic weighting. ′ for:

[0124] F ′ =W k ·(A cd ⊙F ′ ][g ).

[0125] As an implementation method, before mapping the weighted features to the three-dimensional grid map, the process further includes: constructing a three-dimensional grid map centered on the host vehicle itself. The specific process is as follows:

[0126] First, determine the construction range of the 3D map based on the actual application scenario and the vehicle's motion range. Set the range to [x gch ,x g[x ]×[y gch ,y g[x ]×[z gch ,z g[x ] to ensure that all spatial areas that the vehicle may come into contact with are covered.

[0127] Next, the 3D space is divided into grids according to a certain interval Δ, and the 3D map is converted into discrete voxels. Set the voxel size to V mcz_ , then the center P of each voxel p The coordinates can be expressed as:

[0128] P p =(x gch +i·Δ,y gch +j·Δ,z gch +k·Δ);

[0129] Where i, j, k are the indices of the voxel on each axis. In this way, the entire 3D space is divided into N voxels.

[0130] For each voxel P p , use the camera's internal and external parameters to map its coordinates back to the image coordinates captured by each camera. Set the camera's internal parameter matrix to K c And the external parameter matrix is ​​K _ , then the voxel coordinate P p The inverse mapping formula is:

[0131]

[0132] Among them, P 2D is the corresponding coordinate of the voxel on the camera image plane. Through the above inverse mapping, the representation of the voxel in the two-dimensional image can be obtained.

[0133] Finally, according to the coordinate P of the voxel on the image plane 2D and weighted features F ′ , get F ′ Mapped to the corresponding feature representation F on the 3D grid map ′ p :

[0134] F ′ p =f(P 2D ,F Pv );

[0135] Among them, f represents the feature F from the image ′ And its coordinates P on the image plane 2D Functions that map to voxel features.

[0136] S3. Process the voxel features through the attention mechanism based on depth information to generate the final feature representation.

[0137] After the features are successfully mapped to 3D space, an attention mechanism based on depth information is used to further determine whether a vehicle exists in the voxel. This mechanism enhances the determination of vehicle presence by comparing the estimated depth value with the baseline depth in the absence of a vehicle.

[0138] As an implementation method, S3 specifically includes:

[0139] S301: Calculate the reference depth in the three-dimensional grid map and the estimated depth value of each single camera, and determine the attention weight combined with the depth information.

[0140] For example, first, calculate the reference depth D \[m_ , which represents the environmental depth information in the absence of a vehicle. Reference depth D \[m_ Expressed as:

[0141]

[0142] Among them, O x , O y , O z are the coordinates of the vehicle origin respectively.

[0143] Then, according to the estimated depth value and the reference depth of each single camera, the attention weight combined with the depth information is determined; the attention weight combined with the depth information is expressed as:

[0144]

[0145] Among them, ∈ is a threshold, when the estimated depth value D P With reference depth D \[m_ When they are consistent, it is considered that there is no vehicle at that location and the weight is assigned to it as 0; when they are inconsistent, it is considered that there is a vehicle at that location and the weight is assigned to it as 1.

[0146] S302: Generate a final feature representation based on the attention weight and voxel features.

[0147] For example, according to the attention weight A combined with depth information ^ The voxel feature F obtained in the previous step ′ p , generate the final feature representation F after deep attention processing p :

[0148] F p =A ^ ·F ′ p .

[0149] S4. Combine 3D convolution and deformable convolution to fuse the final feature representation of multiple views, generate the occupancy of the 3D grid and determine whether there is a vehicle in the 3D grid.

[0150] In order to make full use of the feature information of each monocular camera and improve the accuracy of vehicle detection, in this embodiment, 3D convolution and deformable 3D convolution are combined to fuse the multi-camera features processed by the attention mechanism.

[0151] As an implementation method, S4 specifically includes:

[0152] S401: Concatenate the final feature representations from each monocular camera to obtain a unified feature representation.

[0153] Let the final feature representation from the i-th monocular camera be F pc , whose dimension is C c ×X c ×Y c ×Z c , C c Represents the number of channels, X c 、Y c and Z c Respectively represent the position of the feature in 3D space. For the final feature representation of n monocular cameras, the stitching operation is expressed as:

[0154]

[0155] Where,

[0156] S402: Use 3D convolution to reduce the dimension of the unified feature representation, and perform deformable 3D convolution on the feature map after the dimension reduction process to obtain a fused feature.

[0157] Among them, the 3D convolution operation is expressed as:

[0158] F ]ihp3D =Conv3D(F ]ih][n ,K);

[0159] Where K is the dimension of C ch ×C ion ×X k ×Y k ×Z k The convolution kernel, C ch and C ion are the number of input and output channels, respectively.

[0160] The deformable 3D convolution operation is expressed as:

[0161]

[0162] Where Δq is the learned offset, W(q) is the convolution kernel weight, and X(p+Δq) is the input feature map F ]ihp3D The value at position p+Δq, is the convolution area.

[0163] S403, use the fully connected layer to fusion feature F ^_`ilg3D Mapped to occupancy probability P i]] , occupation probability P i]] Expressed as:

[0164] P i]] =σ(W·F ^_`ilg3D +b);

[0165] Where W is the weight matrix of the fully connected layer, b is the bias term, and σ is the Sigmoid activation function, which is used to output the occupancy probability, ranging from 0 to 1.

[0166] S404: Based on the output occupancy probability P i]] Determine occupancy in a 3D grid.

[0167] Specifically, the threshold θ is set, and the occupancy state O of the voxel v is p Expressed as:

[0168]

[0169] In the formula, 1 means it is occupied and 0 means it is not occupied.

[0170] In order to improve the generation stability of the three-dimensional occupancy grid and reduce the detection instability caused by instantaneous occlusion or view angle changes, in some embodiments, the following steps are further included:

[0171] S5. Obtain a three-dimensional occupancy grid sequence; process the three-dimensional occupancy grid sequence through the trained time series feature extraction network, obtain the occupancy probability after time series fusion, and determine whether there is a vehicle.

[0172] In this embodiment, the temporal feature extraction network is LSTM (Long Short-Term Memory Network), and the input is the generated three-dimensional occupancy grid sequence in, Represents the occupancy grid of the tth frame. Each occupancy grid contains a channel C (such as vehicle, background) and the position coordinates (x, y, z) in three-dimensional space. The output is the occupancy grid probability O after time series information fusion. ′ n . Occupancy probability O ′ n Expressed as:

[0173] O ′ n =fLSTM ({O n-h ,O n-h+1 ,…,O n}).

[0174] In practical applications, this 3D grid not only contains object detection information from the current frame but also incorporates historical information from previous frames, thereby reducing error fluctuations caused by perspective changes or transient occlusions. Therefore, this embodiment uses an LSTM to process the 3D occupancy grid sequence. By memorizing and updating feature information from previous moments, the LSTM can effectively capture dynamic changes between consecutive frames.

[0175] Finally, by setting the threshold τ, the occupied grid after time series fusion is judged to determine whether there is a vehicle:

[0176]

[0177] Based on this, by modeling the historical information of continuous frames, the motion trajectory of the object is smoothed to ensure the accuracy of vehicle detection in dynamic scenes.

[0178] In the multi-camera feature fusion and temporal information fusion parts, in this embodiment, a loss function based on occupancy probability is designed.

[0179] Let the generated occupancy grid be O ion (x,y,z), whose value represents the probability of whether the coordinate (x,y,z) in the three-dimensional grid is occupied by a vehicle. The real label grid is Its value is 0 or 1, indicating whether the location is occupied by a vehicle. The binary cross entropy loss is used to measure the difference between the predicted grid and the true grid:

[0180]

[0181] At the same time, in order to constrain the smoothness between consecutive frames and reduce the sudden change of occupancy state caused by viewpoint changes or occlusion, the following temporal smoothing loss is used:

[0182]

[0183] Where n represents smooth iteration of n consecutive frames of output, N is the total number of voxels in the output 3D grid map, represents the predicted occupancy state of the i-th voxel at time step t.

[0184] The total loss combines the temporal occupancy classification loss and the temporal smoothing loss:

[0185]

[0186] Among them, λ1 and λ2 are the weight coefficients of the two loss terms, which are used to balance the occupancy classification loss and the temporal smoothing loss.

[0187] Next, the effectiveness of the vehicle detection method based on the bionic compound eye array camera described in this embodiment is further verified by combining simulation experiments.

[0188] The simulation environment used in this example is CARLA Simulator, an open-source autonomous driving simulator designed specifically for autonomous driving research in urban environments. It provides a highly realistic virtual environment encompassing dynamic elements such as urban traffic, road infrastructure, and vehicles. It also supports the simulation of multiple sensors, such as RGB cameras, semantic segmentation cameras, depth cameras, and radar, enabling accurate simulation of the vehicle's surroundings. CARLA also provides a flexible API interface for programmatic interaction with the simulator, enabling the configuration of custom scenarios, autonomous driving tasks, and various environmental conditions.

[0189] In the simulation experiment, an automatic data acquisition system was designed. The system is divided into two parts:

[0190] Data acquisition module:

[0191] This module is responsible for acquiring multimodal data about the vehicle environment from the CARLA simulator. By connecting to the CARLA simulator, the module collects RGB images, semantically segmented images, depth images, and related vehicle position information in real time, providing high-quality data support for subsequent vehicle detection and 3D reconstruction tasks. This module is specifically designed for bionic compound eye array camera systems. By synchronously collecting data from multiple perspectives, it ensures multimodal information fusion across a wide field of view. The data acquisition process is as follows:

[0192] First, the system connects to the CARLA simulator server to retrieve world objects from the simulation environment. The data acquisition module then stores the collected RGB images, semantically segmented images, depth images, and vehicle information in separate output folders according to predefined paths and output formats. Separate output directories are created for RGB and semantically segmented images, and files are named using the simulation frame number to ensure data order.

[0193] Data acquisition primarily consists of two components: Image acquisition: This module periodically extracts RGB images from the bionic compound eye array camera in the CARLA simulator, and simultaneously acquires semantically segmented and depth images from the camera. These images are then saved in PNG format to the local hard drive and serve as input for subsequent feature extraction and depth estimation. Information acquisition: In addition to images, the module also collects vehicle location information and corresponding semantic segmentation labels. This information is stored in a CSV file to facilitate subsequent data analysis and feature annotation.

[0194] During data acquisition, the system synchronously collects image information and position information to provide a basis for the subsequent fusion stage. The sampling time interval is set to T, and the data collected in each frame is recorded as:

[0195]

[0196] in, represents an RGB image, is the semantic segmentation image, is the depth image, P c is the vehicle’s location information. The system forms a dataset D with all the frame data. c , used for subsequent feature extraction and deep learning model training.

[0197] At the same time, the vehicle's size information and semantic color information are also collected. The vehicle's size is used to construct the ground truth of the 3D mesh map, and the vehicle's semantic color information is used for further processing of the semantic segmentation image.

[0198] Data consistency verification and preprocessing module:

[0199] This module's primary function is to perform consistency checks and preprocessing on the collected data, ensuring that the multimodal data is intact after acquisition and stored and organized in a structured manner for subsequent model training and analysis. The module first checks whether the number of images in each output folder is consistent with the expected number to prevent image loss due to interruptions or anomalies. The module then performs appropriate preprocessing based on the path information. Preprocessing primarily consists of the following steps:

[0200] Semantic Segmentation Image Preprocessing: This component preprocesses the semantically segmented image, aiming to extract specific vehicle regions from the original image and remove the segmentation information of the ego vehicle for subsequent analysis and vehicle detection. First, the target vehicle region is extracted from the semantically segmented image based on the preset color range for vehicles in the CARLA Simulator. By converting the image to the HSV color space, the module uses a set of preset color ranges to identify vehicles in the semantically segmented image. Simultaneously, the module identifies the color information of the ego vehicle acquired during data acquisition. By removing the color region of the ego vehicle from the semantically segmented image, only the semantic information of other vehicles is retained. After processing, the generated semantic information is saved in a 0-1 mask format.

[0201] Depth Image Preprocessing: After extracting depth values, the module processes them according to the specified depth type. This process includes depth value conversion and data normalization, ensuring that the resulting depth data accurately reflects the positions of objects in the scene. During processing, the module first converts the RGB channel values ​​into depth values ​​using the depth decoding formula provided by the CARLA Simulator. The depth value calculation formula is as follows:

[0202]

[0203] Where R, G, and B represent the pixel values ​​of the red, green, and blue channels in the image, respectively. This calculation performs a weighted summation of the information in the RGB channels to obtain the actual depth value, which ranges from 0 to 1000 meters. Due to the small perception range of the present invention, we truncate depth values ​​greater than 100 meters to allow the model to focus more on the effective depth range. That is:

[0204]

[0205] After extracting the depth values, the module processes them according to the specified depth type. For depth types, the depth values ​​are normalized to the range [0, 1]. For logarithmic depth types, the data is further converted to logarithmic form. This processing step is designed to enhance the usability of the depth data, laying the foundation for subsequent depth estimation and analysis. After preprocessing, the generated depth mask is saved in NumPy array format.

[0206] 3D grid map truth value construction: Since the main purpose of this embodiment is to achieve vehicle detection by constructing a 3D grid map, it is necessary to obtain the truth value of the 3D grid map. First, it is necessary to define the size parameters of the 3D grid map, including the length, width and height size settings, as well as the size of each unit interval. Then, by reading the CSV file containing vehicle size information, the module extracts key parameters such as the length, width and height of different types of vehicles. Here we use the drawing function of matplotlib to generate the truth value. The specific steps are: for each height layer, use a rectangular box to draw the outline of each vehicle in the figure according to the size and position of the vehicle. After completion, set the range of the coordinate axis and save the intermediate results. The generated image is then post-processed, including converting it to a grayscale image, removing edges and binarization to construct it into a 2D occupancy grid map. Finally, the images are spliced ​​at the height layer and saved in NumPy array format to complete the truth value construction of the 3D grid map.

[0207] Dataset collection:

[0208] To meet the needs of vehicle detection tasks, based on the above-mentioned dataset acquisition system, through refined scene settings and parameter configuration, we ensure that the collected multi-view, multi-frame data has high temporal and spatial consistency and coverage, providing comprehensive support for subsequent model training and verification.

[0209] Parameter settings for data collection

[0210] The data set is collected using precise parameter configuration to ensure high-quality collection of each frame of data. The specific parameters are as follows:

[0211] SLR camera parameters: 120x120 pixels, field of view 60°.

[0212] Array camera structure: A total of 12 single-lens cameras are used, arranged in three layers of parallel staggered arrangement, with an overall field of view of approximately 180°.

[0213] Time interval: The frame rate of data acquisition was set to 10 frames / s.

[0214] Collection range: The scene of the collection system covers a space of 50m x 100m x 2m, with a grid interval of 0.5m.

[0215] This example collects 20 different scene data segments, each lasting 7 to 20 seconds, and the overall data size exceeds 2,000 frames. In order to ensure the diversity of the data set, the present invention collects a variety of different scenes, including:

[0216] Road scenes: include straight roads, curves, intersections and other road conditions, simulating vehicle driving and parking scenes in urban traffic.

[0217] Lighting environment: Simulates poor lighting conditions to test the robustness of vehicle detection in low-light environments such as early morning or dusk.

[0218] Weather conditions: Introduce weather conditions such as rainy days and foggy days to test the system's perception performance under different weather conditions.

[0219] Evaluation indicators:

[0220] In this example, the intersection-over-union (IoU) ratio is used as the primary evaluation metric to measure the model's accuracy in detecting vehicle foreground areas in a 3D grid. IoU measures the degree of overlap between the predicted and true areas and is defined as the ratio of the intersection to the union of the predicted occupied area and the true occupied area. The formula is:

[0221]

[0222] Among them, V jl_^ represents the three-dimensional occupied area predicted by the model, V an Represents the true three-dimensional occupied area.

[0223] Specifically for the task in this example, Intersection over Union (IoU) is used to evaluate the detection of vehicle occupancy areas within the model-generated 3D occupancy grid. Since the background area is not a core objective in this task, we only calculate the IoU with the vehicle foreground area to measure the model's accuracy in predicting vehicle occupancy. A high IoU value indicates that the model's predicted vehicle occupancy areas are highly consistent with the true area, effectively capturing the presence of the vehicle and its position in 3D space.

[0224] Experimental results:

[0225] Under the above evaluation indicators, the IOU of the final output result of the model after training on the dataset is 0.4613.

[0226] Example 3

[0227] This embodiment discloses a vehicle detection system based on a bionic compound eye array camera, comprising:

[0228] A feature extraction module is configured to: obtain single-camera images from multiple perspectives, perform feature extraction on each of the multiple single-camera images, and generate corresponding camera features and semantic segmentation prediction results;

[0229] The feature generation module is configured to: process the semantic segmentation prediction results through an attention mechanism based on semantic information and map them to a three-dimensional grid map to obtain voxel features; process the voxel features through an attention mechanism based on depth information to generate a final feature representation;

[0230] The multi-camera feature fusion module is configured to fuse the final feature representations of multiple perspectives by combining 3D convolution and deformable convolution, generate the occupancy of the 3D grid, and determine whether there is a vehicle in the 3D grid.

[0231] It should be noted that the feature extraction module, feature generation module, and multi-camera feature fusion module described above correspond to the steps in Example 1. The examples and application scenarios implemented by these modules and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0232] Example 4

[0233] Embodiment 4 of the present invention provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps of the above-mentioned vehicle detection method based on the bionic compound eye array camera are completed.

[0234] Example 5

[0235] A fifth embodiment of the present invention provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps of the above-mentioned vehicle detection method based on the bionic compound eye array camera are completed.

[0236] Example 6

[0237] A sixth embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned vehicle detection method based on a bionic compound eye array camera.

[0238] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0239] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0240] These computer program instructions can also be loaded onto a computer or other programmable data processing device, and a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide the functions for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0241] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0242] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A bionic compound eye array camera system, characterized in that: Includes a multi-layer SLR camera and a spherical mount; Multiple layers of SLR cameras are arranged on the spherical bracket along a longitudinal spacing, and adjacent layers of SLR cameras are arranged in parallel and staggered manner; Each layer of SLR cameras includes a plurality of SLR cameras, and the plurality of SLR cameras are arranged on the spherical bracket along a lateral interval.

2. A vehicle detection method based on a bionic compound eye array camera, characterized in that: include: Acquire single-camera images from multiple perspectives, perform feature extraction on multiple single-camera images respectively, and generate corresponding camera features and semantic segmentation prediction results; wherein the single-camera images from multiple perspectives are collected by a bionic compound eye array camera system; The camera features and semantic segmentation prediction results are processed and mapped to a three-dimensional grid map through an attention mechanism based on semantic information to obtain voxel features. The voxel features are processed through an attention mechanism based on depth information to generate the final feature representation. The final feature representations of multiple views are fused by combining 3D convolution and deformable convolution to generate the occupancy of the 3D grid and determine whether there is a vehicle in the 3D grid.

3. The vehicle detection method based on the bionic compound eye array camera according to claim 1, characterized in that: Also includes: Get a three-dimensional occupancy grid sequence; The three-dimensional occupancy grid sequence is processed through the trained time series feature extraction network to obtain the occupancy probability after time series fusion and determine whether there is a vehicle.

4. The vehicle detection method based on the bionic compound eye array camera according to claim 1, characterized in that: The feature extraction of multiple single-camera images respectively to generate corresponding camera features and semantic segmentation prediction results specifically includes: The Conv-UNext network is used to extract features of multiple single-camera images in turn to obtain the corresponding camera features; the camera features are processed by depth-separable convolution and Sigmoid activation function in turn to generate semantic segmentation prediction results.

5. The vehicle detection method based on the bionic compound eye array camera according to claim 1, characterized in that: The camera features and semantic segmentation prediction results are processed and mapped to a three-dimensional grid map by an attention mechanism based on semantic information to obtain voxel features, specifically including: Multiply the camera features by the corresponding semantic segmentation prediction results element by element to obtain the intermediate feature representation; Calculate the foreground ratio of each monocular camera, and determine the attention weight of each monocular camera according to the foreground ratio; obtain weighted features according to the attention weight and intermediate feature representation; According to the coordinates and weighted features of each voxel in the three-dimensional grid map on the image plane, the corresponding voxel features are obtained.

6. The vehicle detection method based on the bionic compound eye array camera according to claim 1, characterized in that: The processing of voxel features by the attention mechanism based on depth information to generate the final feature representation specifically includes: Calculate the baseline depth of the 3D grid map and the estimated depth value of each single camera, and determine the attention weight that combines the depth information; Based on the attention weights and voxel features, the final feature representation is generated.

7. The vehicle detection system based on the bionic compound eye array camera is characterized by: include: The feature extraction module is configured to: obtain single camera images of multiple perspectives, perform feature extraction on multiple single camera images respectively, and generate corresponding camera features and semantic segmentation prediction results; The feature generation module is configured to: process the semantic segmentation prediction results through an attention mechanism based on semantic information and map them to a three-dimensional grid map to obtain voxel features; process the voxel features through an attention mechanism based on depth information to generate a final feature representation; The multi-camera feature fusion module is configured to fuse the final feature representations of multiple perspectives by combining 3D convolution and deformable convolution, generate the occupancy of the three-dimensional grid and determine whether there is a vehicle in the three-dimensional grid.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the vehicle detection method based on the bionic compound eye array camera according to any one of claims 2 to 6.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps of the vehicle detection method based on the bionic compound eye array camera described in any one of claims 2 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the steps of the vehicle detection method based on the bionic compound eye array camera described in any one of claims 2 to 6 are implemented.

Images