Vehicle detection method and system based on biomimetic compound eye array camera

Abstract

The application discloses a vehicle detection method based on a bionic compound eye array camera and belongs to the technical field of automatic driving. The method comprises the following steps: acquiring single-camera images of multiple perspectives, respectively extracting features from the multiple single-camera images, and generating corresponding camera features and semantic segmentation prediction results; processing and mapping the camera features and the semantic segmentation prediction results to a three-dimensional grid map through a semantic information-based attention mechanism to acquire voxel features; processing the voxel features through a depth information-based attention mechanism to generate final feature representations; combining 3D convolution and deformable convolution to fuse the final feature representations of the multiple perspectives, generating an occupancy situation of the three-dimensional grid, and judging whether a vehicle exists in the three-dimensional grid. The method can fuse multi-perspective information of an overlapping area of cameras, and solves the problem that the detection accuracy of vehicle perception and detection under a large field of view needs to be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a vehicle detection method and system based on a bionic compound eye array camera. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The research of bionic compound eye array cameras began with the in-depth observation and imitation of insect visual systems, especially the superior performance of insect compound eyes in wide field of view, fast response and multi-target tracking. Insect compound eyes are composed of a large number of simple light sensing units (called "ommatidia"), each of which has a fixed viewing angle and overlaps part of the viewing field with each other. By integrating this information, insects can quickly perceive the movement of objects in a wide range. Based on these characteristics, the compound eye structure has inspired many scholars to study bionic visual systems.

[0004] Early research on bionic compound eye cameras mainly focused on simulating the wide field of view characteristics of insect vision. With the construction of the mathematical model of compound eye imaging, the geometric characteristics and optical perception principles of insect compound eyes gradually became known to people. Subsequently, scientists used this model to construct a bionic camera system with wide field of view perception capability. At the same time, with the development of microelectronic manufacturing technology, a wide field of view and high sensitivity bionic compound eye camera system was manufactured, and by integrating multiple low-resolution sensors into the same device, a basic model of bionic compound eye array camera was constructed.

[0005] With the wide application of bionic compound eye cameras, how to effectively fuse the overlapping fields of view of multiple low-resolution cameras has become an important research direction. Most current research focuses on aligning and fusing the overlapping field of view information captured by different cameras to obtain a wider field of view. That is, by splicing the images between the array cameras, a large field of view angle image is generated, and on this basis, target detection and other tasks in a large field of view environment are realized.

[0006] Vehicle detection is a key task in applications such as autonomous driving, intelligent transportation, and urban monitoring. With the progress of computer vision, vehicle detection technology has developed from 2D image target detection to current 3D target detection. 3D target detection not only provides more accuracy in the spatial positioning of vehicles, but also provides a basis for predicting the behavior of objects, route planning, and environmental understanding. Occupancy Network (Occ) is a new technology for 3D target detection proposed in recent years, which works by mapping input data from multiple view images into a three-dimensional feature space, and then learning the occupancy probability of each point in the feature space through a deep neural network.

[0007] However, the existing Occ has high demand for computing resources, and it is difficult to effectively utilize the overlapping information between cameras under the conventional camera layout specification. SUMMARY

[0008] To solve the problems of the prior art, the present application provides a bionic compound eye array camera system, a vehicle detection method, system, electronic device, computer readable storage medium and computer program product based on the bionic compound eye array camera, which realizes vehicle perception and detection under a large field of view by innovative application of the bionic compound eye array camera, deep fusion of multi-view information, and introduction of semantic and depth attention mechanisms, and improves the accuracy of vehicle detection.

[0009] In a first aspect, the present application provides a bionic compound eye array camera system;

[0010] The bionic compound eye array camera system comprises a plurality of layers of single-eye cameras and a spherical support;

[0011] The plurality of layers of single-eye cameras are arranged at longitudinal intervals on the spherical support, and adjacent layers of single-eye cameras are arranged in parallel and staggered.

[0012] Each layer of single-eye cameras comprises a plurality of single-eye cameras arranged at lateral intervals on the spherical support.

[0013] In a second aspect, the present application provides a vehicle detection method based on the bionic compound eye array camera;

[0014] The vehicle detection method based on the bionic compound eye array camera comprises:

[0015] Obtaining multi-view single-camera images, performing feature extraction on the plurality of single-camera images respectively to generate corresponding camera features and semantic segmentation prediction results; wherein the multi-view single-camera images are collected by the bionic compound eye array camera system;

[0016] Processing and mapping the camera features and the semantic segmentation prediction results to a three-dimensional grid map through an attention mechanism based on semantic information to obtain voxel features; processing the voxel features through an attention mechanism based on depth information to generate a final feature representation;

[0017] Fusing the multi-view final feature representations through 3D convolution and deformable convolution to generate an occupancy situation of the three-dimensional grid and determine whether a vehicle exists in the three-dimensional grid.

[0018] In some embodiments, the method further comprises:

[0019] Obtaining a three-dimensional occupancy grid sequence;

[0020] The trained time sequence feature extraction network is used for processing the three-dimensional occupation grid sequence, occupation probability after time sequence fusion is obtained, and it is judged whether a vehicle exists.

[0021] In some embodiments, the feature extraction on the plurality of single-camera images respectively, and generating corresponding camera features and semantic segmentation prediction results specifically comprises:

[0022] The Conv-UNext network is used for sequentially extracting features from the plurality of single-camera images, and the corresponding camera features are obtained; the camera features are sequentially processed by depth separable convolution and Sigmoid activation function, and semantic segmentation prediction results are generated.

[0023] In some embodiments, the camera features and the semantic segmentation prediction results are processed and mapped to the three-dimensional grid map by the attention mechanism based on semantic information, and the voxel features are obtained specifically comprising:

[0024] The camera features are multiplied with the corresponding semantic segmentation prediction results element by element to obtain intermediate feature representations;

[0025] The foreground proportion of each monocular camera is calculated, the attention weight of each monocular camera is determined according to the foreground proportion, and the weighted features are obtained according to the attention weight and the intermediate feature representations.

[0026] According to the coordinates of each voxel in the three-dimensional grid map on the image plane and the weighted features, the corresponding voxel features are obtained.

[0027] In some embodiments, the voxel features are processed by the attention mechanism based on depth information to generate the final feature representation specifically comprising:

[0028] The reference depth of the three-dimensional grid map and the estimated depth value of each single camera are calculated, and the attention weight combined with the depth information is determined;

[0029] The final feature representation is generated according to the attention weight and the voxel features.

[0030] In a third aspect, the present application provides a vehicle detection system based on a bionic compound eye array camera.

[0031] The vehicle detection system based on the bionic compound eye array camera comprises:

[0032] The feature extraction module is configured to obtain multi-view single-camera images, extract features from the plurality of single-camera images respectively, and generate corresponding camera features and semantic segmentation prediction results.

[0033] The feature generation module is configured to: acquire voxel features by processing and mapping the semantic segmentation prediction result to a three-dimensional grid map based on a semantic information attention mechanism; and generate a final feature representation by processing the voxel features based on a depth information attention mechanism.

[0034] The multi-camera feature fusion module is configured to: combine 3D convolution and deformable convolution to fuse the multi-view final feature representations, generate an occupancy situation of the three-dimensional grid, and determine whether a vehicle exists in the three-dimensional grid.

[0035] In a fourth aspect, the present application provides an electronic device.

[0036] An electronic device includes a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the vehicle detection method based on the biomimetic compound eye array camera.

[0037] In a fifth aspect, the present application provides a computer readable storage medium.

[0038] A computer readable storage medium has a computer program / instruction stored thereon, and the computer program / instruction is executed by a processor to implement the steps of the vehicle detection method based on the biomimetic compound eye array camera.

[0039] In a sixth aspect, the present application provides a computer program product.

[0040] A computer program product includes a computer program / instruction, and the computer program / instruction is executed by a processor to implement the steps of the vehicle detection method based on the biomimetic compound eye array camera.

[0041] Compared with the prior art, the present application has the following beneficial effects:

[0042] 1. The technical solution provided by the present application realizes efficient perception of a large field of view through a biomimetic compound eye array camera system, effectively simulates the wide field of view advantage of an insect compound eye, significantly improves the multi-view information capture capability of the detection system in a large range, makes the vehicle detection coverage more extensive, and can meet the panoramic perception demand in a complex environment.

[0043] 2. The technical solution provided by the present application introduces a double attention mechanism based on semantic information and depth information, realizes precise perception of a multi-level attention mechanism, makes the detection method focus on a key vehicle area, effectively suppresses background interference, first weights the vehicle area by using semantic information in feature processing, then further accurately determines whether a vehicle exists by using depth information, and realizes high-precision perception in a three-dimensional space.

[0044] 3、The technical scheme provided by the present application realizes the improvement of multi-camera information fusion, adopts the combination of 3D convolution and deformable 3D convolution, so that the features in the overlapping area of each camera can be better aligned and fused, and the comprehensive expression ability of multi-view features is significantly enhanced. At the same time, through the strategy of residual connection and feature dimension reduction, the calculation burden is effectively reduced and the reliability of the multi-camera fusion result is maintained, ensuring the accuracy of the occupancy in the three-dimensional space.

[0045] 4、The technical scheme provided by the present application enhances the stability of vehicle detection through time sequence information fusion, can smooth the motion change of objects between consecutive frames, reduces the detection instability caused by instantaneous occlusion or view angle change, ensures the stability and accuracy of vehicle tracking and detection in dynamic scenes, and provides higher reliability for complex scene perception in actual application. BRIEF DESCRIPTION OF DRAWINGS

[0046] The drawings accompanying the specification of the present application form a part thereof, serve to provide further understanding of the present application, and together with the description of the exemplary embodiments of the present application and the explanation thereof serve to explain the present application, and do not constitute an improper limitation of the present application.

[0047] Figure 1 The flowchart of the vehicle detection method provided by the embodiment of the present application is shown in the figure;

[0048] Figure 2 The network architecture diagram of the Conv-UNext encoder provided by the embodiment of the present application is shown in the figure;

[0049] Figure 3 The flowchart of data acquisition in the simulation experiment provided by the embodiment of the present application is shown in the figure;

[0050] Figure 4 The flowchart of data preprocessing in the simulation experiment provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] It should be pointed out that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present application have the same meaning as generally understood by those skilled in the art to which the present application belongs.

[0052] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the exemplary embodiments of this disclosure is intended to be covered no matter how it is expressed. Use of the alternative (e.g. "or") should be understood to mean either term or both terms. For example, if a condition is X or Y, then the condition is satisfied if any one of X is true or Y is true. As used herein, singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," or "includes" and / or "including" when used in this specification, specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof.

[0053] Embodiments in the present disclosure and features in the embodiments can be combined with each other if there is no conflict.

[0054] Embodiment One

[0055] The bionic compound eye array design realizes all-around visual field coverage, can capture environmental information in a large range around the vehicle, and is particularly suitable for detection tasks of complex targets in dynamic scenes. Compared with traditional monocular cameras or non-overlapping multi-view cameras, the compound eye array significantly improves the ability to capture environmental details.

[0056] The present embodiment provides a bionic compound eye array camera system, which realizes the collection of multi-view information by simulating the multi-view visual characteristics of insect compound eyes. The system aims to cover a wider viewing angle and capture the environment in detail, and has a significant advantage in vehicle detection tasks.

[0057] An insect compound eye is composed of multiple functionally independent ommatidia, each ommatidium has a limited field of view. Through the joint layout of ommatidia, the insect compound eye can realize wide visual coverage and rapidly perceive targets in dynamic environments. The bionic compound eye array camera system provided by the present embodiment includes a multi-layer ommatidium camera and a spherical support, the multi-layer ommatidium camera is installed on the spherical support along the longitudinal direction, and adjacent layers of ommatidium cameras are arranged in parallel. Each layer of ommatidium camera includes multiple ommatidium cameras, and the multiple ommatidium cameras are installed on the spherical support along the transverse direction. Through the multi-layer parallel staggered arrangement, the optimal design of the field of view coverage is realized.

[0058] As an embodiment, the bionic compound eye array camera system for providing multi-view information includes 3 layers of ommatidium cameras, adopts a three-layer parallel staggered arrangement, and the number of ommatidium cameras in each layer from top to bottom is 6, 5, and 6 respectively. Specifically, each ommatidium camera adopts a low-resolution RGB camera, the resolution of the ommatidium camera is 120x120 pixels, the field of view angle is 60°, the ommatidium cameras are distributed on a spherical support with a diameter of 20 cm, the included angle of the ommatidium cameras in each layer is 30°, and the included angle between layers is 15°.

[0059] Under this condition, the perception of 50m in front, left and right directions of the vehicle can be realized, and in this range, any position is covered by at least 3 single-eye cameras. At the same time, this arrangement can easily realize the increase and decrease of the number of cameras to achieve different range of perception size.

[0060] This arrangement can capture important information in the environment while reducing data redundancy and computational overhead. The compound eye array system can provide sufficient overlapping areas while maintaining a wide viewing angle. Information fusion in the overlapping area can effectively improve the accuracy of vehicle detection, especially in complex scenes around the vehicle. This embodiment realizes efficient and comprehensive information collection through the biomimetic compound eye array camera system, laying a solid foundation for subsequent multi-camera information fusion, feature extraction and dynamic detection in complex scenes. This design fully utilizes the advantages of bionics, combines the wisdom of nature with artificial vision systems, and provides an innovative solution for the field of autonomous driving.

[0061] Embodiment two

[0062] The existing vehicle detection method lacks the use of overlapping information between cameras, resulting in the need to improve the accuracy and efficiency of vehicle detection in dynamic scenes. Therefore, the present application provides a vehicle detection method based on a biomimetic compound eye array camera, focusing on vehicle detection based on a biomimetic compound eye array camera, and realizing vehicle perception and detection under a large field of view by fusing multi-view information in the overlapping area of the camera.

[0063] Next, combined with Figures 1-4 A vehicle detection method based on a biomimetic compound eye array camera is disclosed in this embodiment. The vehicle detection method based on the biomimetic compound eye array camera includes the following steps:

[0064] S1, acquire multi-view single camera images, and respectively extract features from the multiple single camera images to generate corresponding camera features and semantic segmentation prediction results.

[0065] As an implementation, S1 specifically includes:

[0066] S101, sequentially extract features from the multiple single camera images by the Conv-UNext network to obtain corresponding camera features.

[0067] In this embodiment, the Conv-UNext network comprises sequentially connected down-sampling layers, four down-convolution blocks, four up-convolution blocks and an up-convolution layer, and the down-convolution blocks are connected in residual manner with the up-convolution blocks; that is, the output of the third down-convolution block is fused with the output of the first up-convolution block and then input into the second up-convolution block, the output of the second down-convolution block is fused with the output of the second up-convolution block and then input into the third up-convolution block, and the output of each down-convolution block is fused with the output of the third up-convolution block and then input into the fourth up-convolution block.

[0068] The down-convolution block comprises sequentially connected 7x7 convolution layers, normalization layers, 1x1 convolution layers, GELU and 1x1 convolution layers, and the output of the 7x7 convolution layer is connected in residual manner with the output of the last 1x1 convolution layer; the up-convolution block comprises sequentially connected 7x7 transpose convolution layers, normalization layers, 1x1 convolution layers, GELU and 1x1 convolution layers, and the output of the 7x7 transpose convolution layer is connected in residual manner with the output of the last 1x1 convolution layer.

[0069] The Conv-UNext network contains designs such as 7x7 window convolution kernel, depth separable convolution and inverse bottleneck structure, which effectively maintains the local receptive field advantage of the convolution network, significantly reduces the computational amount through large convolution kernel and simplified activation function / normalization layer setting, and enhances the training stability and robustness of the network.

[0070] Specifically, in the encoder of the Conv-UNext network, first, the input monocular image is down-sampled, and the size of the image is adjusted through Layer Normalization (LN) and convolution layer.

[0071] For example, the size of the input monocular image is HxW, and the number of feature channels is C, then the size of the feature map after down-sampling can be represented as:

[0072]

[0073] Wherein, X is the monocular image, Conv(·) represents the convolution operation, H and W are the height and width of the monocular image respectively.

[0074] In each subsequent Block, in order to reduce the computational amount, the dimension lifting ratio of the inverse bottleneck structure is reduced from the original 4 times to 2 times in this embodiment. The depth separable convolution is further used to reduce the model parameter amount, while still maintaining a large convolution kernel size (7x7) to expand the receptive field and improve the efficiency of feature extraction. The output of the depth separable convolution in the Block can be represented as:

[0075]

[0076]

[0077] wherein, DepthwiseConv(·) represents a depthwise separable convolution, and PointwiseConv(·) represents a pointwise convolution.

[0078] In the decoder part, in order to maintain symmetry with the structure of the encoder and at the same time enhance the ability of feature upsampling, a 7x7 transpose convolution kernel is used in this embodiment, which can reduce information loss while enlarging the size of the feature map.

[0079] Suppose the size of the input feature map is H / 4xW / 4, after the transpose convolution, it is restored to the original size:

[0080]

[0081] At this point, the camera feature F of the input single camera image is extracted.

[0082] S102, the camera feature is sequentially processed by depthwise separable convolution and Sigmoid activation function, to generate a semantic segmentation prediction result.

[0083] In order to further improve the detection performance of the model, on the basis of feature extraction, the semantic information and depth information in the image are further processed. By using a large convolution kernel and depthwise separable convolution to reduce the dimension of the feature, and combining the Sigmoid function for normalization, the final semantic segmentation prediction and depth segmentation prediction results are obtained.

[0084] For example, suppose the camera feature is The output feature map after depthwise separable convolution dimension reduction processing is:

[0085]

[0086] Next, two detection heads are used to process the output feature map respectively to extract semantic information and depth information, that is, to generate specific semantic feature map S m_a and depth feature map S ^_jnb :

[0087]

[0088] wherein, SegConv and DepthConv represent semantic detection head and depth detection head respectively.

[0089] Then, through the Sigmoid activation function, the semantic segmentation prediction result P m_a and the depth segmentation prediction result P ^_jnb :

[0090] P m_a = σ(Sm_a );

[0091] P ^_jnb =σ(S ^_jnb ).

[0092] In this step, the model is trained using a semantic segmentation loss function and a depth estimation loss function during the training process.

[0093] For the semantic segmentation part, this embodiment uses Dice Loss as the loss function to measure the matching degree between the model prediction and the true label. Dice Loss is expressed as:

[0094]

[0095] Where, p c ,g c Let represent the predicted value and the true label of the i-th pixel, respectively, and N be the total number of pixels.

[0096] By using Dice Loss, this real-time solution can better optimize semantic segmentation tasks, especially when the target region is small. Dice Loss can effectively avoid class imbalance and ensure more accurate segmentation results.

[0097] To improve the accuracy of depth estimation, this embodiment employs a multi-camera joint depth estimation method. By fusing depth information from multiple camera perspectives, the consistency of depth estimation is optimized and errors are reduced. Assume N cameras simultaneously capture the same scene. For a given real-world 3D coordinate point P in the scene, each camera calculates the depth estimate for that point. The depth estimation loss consists of two parts: an absolute error-weighted loss and a consistency loss.

[0098] The absolute error loss directly compares the depth estimates d from each camera. c (P) and the true depth d GT The difference in (P) ensures that the depth estimated by each camera gradually approximates the true value. The definition is as follows:

[0099]

[0100] Among them, w c It is the confidence weight of camera i, d c (P) is P ^_jnb The depth estimate of point P by camera i.

[0101] The consistency loss constraint aims to ensure that the depth estimates from multiple cameras are as consistent as possible at the same ground truth coordinate point P, reducing estimation discrepancies between cameras. For the depth estimates d from two different cameras i and j... c (P) and d d(P), the consistency loss is defined as:

[0102]

[0103] This loss measures the difference in depth estimation between different cameras, ensuring that the estimates remain consistent across multiple perspectives, thereby improving the overall accuracy of depth estimation.

[0104] The final total depth estimation loss function combines the depth estimation error with the consistency error, weighted by the confidence:

[0105]

[0106] where λ1 and λ2 are the weight coefficients of the two loss terms, used to balance the depth estimation error and consistency optimization.

[0107] To ensure the effectiveness of the optimization, the weight w c The depth estimation confidence of each camera at a specific perspective is obtained by analyzing the depth consistency with surrounding cameras and the internal estimation accuracy of the current camera. The depth estimation value D P of the optimized real coordinate point P

[0108]

[0109] S2, by processing and mapping the camera features and semantic segmentation prediction results to the three-dimensional grid map based on the attention mechanism based on semantic information, obtaining voxel features.

[0110] Before mapping the features to the three-dimensional grid map, in this embodiment, a semantic-based attention mechanism is introduced to enhance the expression ability of the features. This mechanism mainly includes two aspects: one is attention processing according to the semantic results, and the other is attention processing for each monocular camera.

[0111] As an implementation, the specific process of processing camera features and semantic segmentation prediction results by the attention mechanism based on semantic information is as follows:

[0112] First, apply pixel-level attention based on semantics to each camera feature, multiply the camera feature and the corresponding semantic segmentation prediction result element by element to obtain the intermediate feature representation.

[0113] For example, let the camera feature be F ][g , the feature F ′ ][g processed by pixel-level attention can be obtained, denoted as:

[0114] F ′ ][g = Pm_a ⊙F ][g ;

[0115] where ⊙ denotes element-wise multiplication.

[0116] Through this formula, it can be ensured that the features of the vehicle region occupy a larger proportion in subsequent processing.

[0117] Then, for each monocular camera, its foreground proportion P ` is calculated, which is defined as the proportion of vehicle pixels in the total pixels in the camera. The calculation formula of the foreground proportion is:

[0118]

[0119] where N is the total number of pixels in a single monocular camera image.

[0120] To achieve the enhancement of the foreground information, the attention weight W k of the camera is defined as the relative weight of its foreground proportion, which is expressed as:

[0121]

[0122] where M is the total number of cameras. In this way, the camera with a higher foreground proportion will obtain a greater attention weight, so that the attention mechanism can effectively focus on the camera containing more vehicle information.

[0123] By combining the above formulas, the weighted feature F ′ obtained by semantic weighting is:

[0124] F ′ =W k ·(A cd ⊙F ′ ][g ).

[0125] As an implementation, before mapping the weighted feature to the three-dimensional grid map, it further includes: constructing a three-dimensional grid map centered on the host vehicle itself. The specific process is as follows:

[0126] First, according to the actual application scenario and the motion range of the vehicle, the construction range of the three-dimensional map is determined. The range is set to [x gch , x g[x ] × [y gch , y g[x ] × [z gch , z g[x ] to ensure that all spatial regions that the vehicle may contact are covered.

[0127] Next, the three-dimensional space is meshed at intervals Δ, and the three-dimensional map is converted into discrete voxels. The voxel size is set to V mcz_ , and the center P p of each voxel can be represented as:

[0128] P p = (x gch +i·Δ, y gch +j·Δ, z gch +k·Δ);

[0129] where i, j, k are the indices of the voxel on each axis. In this way, the entire three-dimensional space is divided into N voxels.

[0130] For each voxel P p , the coordinates are back-projected to the image coordinates captured by each camera using the intrinsic and extrinsic parameters of the camera. The intrinsic matrix of the camera is set to K c and the extrinsic matrix is set to K _ . The back-projection formula of the voxel coordinates P p is:

[0131]

[0132] where P 2D is the corresponding coordinate of the voxel on the camera image plane. Through the above back-projection, the representation of the voxel in the two-dimensional image can be obtained.

[0133] Finally, according to the coordinates P 2D of the voxel on the image plane and the weighted features F ′ , the F ′ is mapped to the corresponding feature representation F ′ on the three-dimensional mesh map. p :

[0134] F ′ p = f(P 2D , F Pv );

[0135] where f represents a function for mapping from the image feature F ′ and its coordinates P 2D on the image plane to the voxel feature.

[0136] S3, the voxel features are processed through an attention mechanism based on depth information to generate the final feature representation.

[0137] After the features are successfully mapped to the three-dimensional space, an attention mechanism based on depth information is used to further determine whether a vehicle exists in the voxel. This mechanism enhances the determination of the existence of a vehicle by comparing the estimated depth value with the baseline depth in the case of no vehicle.

[0138] As an implementation, S3 specifically includes:

[0139] S301, calculate the baseline depth in the three-dimensional grid map and the estimated depth value of each single camera, and determine the attention weight combined with the depth information.

[0140] For example, first, the baseline depth D \[m_ , which represents the depth information of the environment in the case of no vehicle, is calculated. \[m_ The baseline depth D x is represented as:

[0141]

[0142] where O y , O z , and O P are the coordinates of the vehicle origin, respectively.

[0143] Then, according to the estimated depth value of each single camera and the baseline depth, the attention weight combined with the depth information is determined; the attention weight combined with the depth information is represented as:

[0144]

[0145] where ∈ is a threshold value, when the estimated depth value D \[m_ is consistent with the baseline depth D P , it is considered that there is no vehicle at this position, and it is given a weight of 0; when it is inconsistent, it is considered that there is a vehicle at this position, and it is given a weight of 1.

[0146] S302, generate the final feature representation according to the attention weight and the voxel feature.

[0147] For example, according to the attention weight A ^ combined with the depth information and the voxel feature F ′ obtained in the previous step, the final feature representation F p after depth attention processing is generated:

[0148] F p = A p · F ^ ′ p .

[0149] S4, combine the final feature representation of multi-view by combining 3D convolution and deformable convolution, generate the occupancy of three-dimensional grid and determine whether there is a vehicle in the three-dimensional grid.

[0150] In order to make full use of the feature information of each monocular camera and improve the accuracy of vehicle detection, in the embodiment, 3D convolution and deformable 3D convolution are combined to fuse the multi-camera features processed by attention mechanism.

[0151] As an implementation, S4 specifically includes:

[0152] S401, splice the final feature representation from each monocular camera to obtain a unified feature representation.

[0153] Let the final feature representation from the i-th monocular camera be F pc , whose dimension is C c ×X c ×Y c ×Z c , C c represents the number of channels, X c , Y c and Z c respectively represent the position of the feature in the 3D space. For the final feature representation of n monocular cameras, the splicing operation is represented as:

[0154]

[0155] In the formula,

[0156] S402, use 3D convolution to reduce the dimension of the unified feature representation, and use deformable 3D convolution on the feature map after dimension reduction to obtain the fused feature.

[0157] Wherein, the 3D convolution operation is represented as:

[0158] F ]ihp3D =Conv3D(F ]ih][n ,K);

[0159] In the formula, K is a convolution kernel with a dimension of C ch ×C ion ×X k ×Y k ×Z k , C ch and C ion are the number of input and output channels respectively.

[0160] The deformable 3D convolution operation is represented as:

[0161]

[0162] wherein Δq is a learned offset, W(q) is a convolution kernel weight, and X(p+Δq) is an input feature map F ]ihp3D at position p+Δq, is a convolution region.

[0163] S403, using a fully connected layer to map the fused feature F ^_`ilg3D to an occupancy probability P i]] , the occupancy probability P i]] is expressed as:

[0164] P i]] =σ(W·F ^_`ilg3D +b);

[0165] wherein W is a weight matrix of the fully connected layer, b is a bias term, and σ is a Sigmoid activation function for outputting the occupancy probability, ranging between 0 and 1.

[0166] S404, judging the occupancy in the three-dimensional grid according to the output occupancy probability P i]] .

[0167] Specifically, a threshold θ is set, and the occupancy state O p of a voxel v is expressed as:

[0168]

[0169] wherein 1 represents occupied, and 0 represents unoccupied.

[0170] To improve the stability of generating the three-dimensional occupancy grid and reduce the detection instability caused by instantaneous occlusion or changes in viewing angle, in some embodiments, the method further comprises:

[0171] S5, obtaining a sequence of three-dimensional occupancy grids; processing the sequence of three-dimensional occupancy grids through the trained time-series feature extraction network, obtaining the occupancy probability after time-series fusion, and judging whether a vehicle exists.

[0172] In the embodiment, the time-series feature extraction network is an LSTM (Long Short-Term Memory network), and the input is a sequence of generated three-dimensional occupancy grids wherein, represents an occupancy grid of the t-th frame, each occupancy grid contains a channel C (such as a vehicle, a background) and a position coordinate (x, y, z) in a three-dimensional space, and the output is an occupancy grid probability O ′ n after time-series information fusion. The occupancy grid probability O ′ n is expressed as:

[0173] O ′ n = fLSTM ({O n-h ,O n-h+1 ,…,O n})。

[0174] In practical applications, the three-dimensional grid not only contains the object detection information of the current frame, but also combines the historical information of the previous frames, thereby reducing the error fluctuations caused by changes in viewing angle or instantaneous occlusion. Therefore, in this embodiment, the LSTM is used to process the three-dimensional occupancy grid sequence. The LSTM can effectively capture the dynamic changes between consecutive frames by memorizing and updating the feature information of the previous time.

[0175] Finally, by setting a threshold τ, the occupancy grid after time series fusion is judged to determine whether a vehicle exists:

[0176]

[0177] Based on this, by modeling the historical information of consecutive frames, the motion trajectory of the object is smoothed, ensuring the accuracy of vehicle detection in dynamic scenes.

[0178] In the multi-camera feature fusion and time series information fusion part, in this embodiment, a loss function based on occupancy probability is designed.

[0179] Let the generated occupancy grid be O ion (x,y,z), whose value represents the probability of whether the coordinates (x,y,z) in the three-dimensional grid are occupied by a vehicle. The real label grid is whose value is 0 or 1, indicating whether the position is occupied by a vehicle. Binary cross-entropy loss is used to measure the difference between the predicted grid and the real grid:

[0180]

[0181] At the same time, in order to constrain the smoothness between consecutive frames and reduce the sudden change of occupancy state caused by changes in viewing angle or occlusion, the following time series smoothing loss is used:

[0182]

[0183] where n represents the smoothing iteration of consecutive n frames of output, N is the total number of voxels in the output 3D grid map, represents the predicted occupancy state of the i-th voxel at time step t.

[0184] The total loss combines the time series occupancy classification loss and the time series smoothing loss:

[0185]

[0186] wherein λ1 and λ2 are weight coefficients of two loss terms, used to balance the occupancy classification loss and the temporal smoothing loss.

[0187] Next, the effectiveness of the vehicle detection method based on the bionic compound eye array camera is further verified by simulation experiments.

[0188] The simulation environment used in this embodiment is CARLA Simulator, an open-source autonomous driving simulator designed for autonomous driving research in urban environments. It provides a highly realistic virtual environment, covering urban traffic, road infrastructure, vehicles, and other dynamic elements, while supporting the simulation of various sensors such as RGB cameras, semantic segmentation cameras, depth cameras, radars, etc., enabling accurate simulation of vehicle surroundings perception. In addition, CARLA provides flexible API interfaces that can be programmed to interact with the simulator, set up custom scenarios, autonomous driving tasks, and various environmental conditions.

[0189] In the simulation experiment, an automated data acquisition system was designed. The system consists of two parts:

[0190] Data acquisition module:

[0191] This module is responsible for obtaining multi-modal data of the vehicle environment from the CARLA simulator. By connecting the CARLA simulator, this module can collect real-time RGB images, semantic segmentation images, depth images, and related vehicle position information, providing high-quality data support for subsequent vehicle detection and three-dimensional reconstruction tasks. This module is specifically designed for the bionic compound eye array camera system, ensuring the fusion effect of multi-modal information under a large field of view through the synchronous collection of multi-view data. The data acquisition process is as follows:

[0192] First, the system connects to the CARLA simulator server to obtain the world objects in the simulation environment. Then, the data acquisition module will store the collected RGB images, semantic segmentation images, depth images, and vehicle information into different output folders according to the predefined path and output format. The system establishes independent output directories for RGB images and semantic segmentation images, and uses simulation frame numbers as file names to ensure the orderliness of the data.

[0193] Data collection mainly contains two parts: image collection: this module periodically extracts the RGB images of the biomimetic compound eye array camera from the CARLA simulator, and synchronously obtains the semantic segmentation images and depth images of the camera. These images are then saved in PNG format on the local hard disk as input for subsequent feature extraction and depth estimation. Information collection: in addition to images, the module also collects the position information of the vehicle and the corresponding semantic segmentation labels. These information are stored through CSV files, which is convenient for subsequent data analysis and feature labeling.

[0194] In data collection, the system synchronously collects image information and position information to provide the basis for the subsequent fusion stage. The sampling time interval is set to T, and the data collected in each frame is denoted as:

[0195]

[0196] Among them, represents the RGB image, is the semantic segmentation image, is the depth image, P c is the position information of the vehicle. The system constructs a data set D c from all frames of data for subsequent feature extraction and depth learning model training.

[0197] At the same time, the size information of the vehicle and the semantic color information of the vehicle are also collected. The size of the vehicle is used to construct the 3D grid map ground truth later, and the semantic color information of the vehicle is used for further processing of the semantic segmentation image.

[0198] Data consistency verification and preprocessing module:

[0199] The main function of this module is to check the consistency of the collected data and to preprocess it, to ensure that the multi-modal data is complete after collection, and to store and organize it in a structured way, which is convenient for subsequent model training and analysis. This module first checks whether the number of images in each output folder is consistent with the expected value, to prevent missing images due to interruption or abnormality. Then, the module will perform corresponding preprocessing according to the path information. Preprocessing mainly includes the following parts:

[0200] Semantic Segmentation Image Preprocessing: This part is responsible for preprocessing the semantic segmentation image, aiming to extract specific vehicle regions from the original image and remove the ego vehicle segmentation information for subsequent analysis and vehicle detection tasks. First, according to the color range preset for vehicles in CARLA Simulator, the target vehicle region is extracted from the semantic segmentation image. By converting the image to HSV color space, the module uses a set of preset color ranges to identify vehicles in the semantic segmentation image. At the same time, the module identifies the color information of the ego vehicle obtained from the data collection process. By removing the color region of the ego vehicle from the semantic segmentation image, only the semantic information of other vehicles is retained. After processing, the generated semantic information is saved in the format of 0-1 mask.

[0201] Depth Image Preprocessing: After completing the depth value extraction, the module processes the depth values according to the specified depth type. The entire process includes depth value conversion and data normalization, ensuring that the final generated depth data accurately reflects the positions of objects in the scene. During processing, first, the module converts the RGB channel values to depth values through the depth decoding formula given by CARLA Simulator. The depth value calculation formula is as follows:

[0202]

[0203] where R, G, and B represent the pixel values of the red, green, and blue channels in the image, respectively. This calculation performs a weighted sum of the information in the RGB channels to obtain the actual depth value, with a range of 0-1000m. Since the perception range of the invention is relatively small, we truncate depth values greater than 100 meters to make the model more focused on the effective depth range. That is:

[0204]

[0205] After completing the depth value extraction, the module processes the depth values according to the specified depth type. For depth type, the depth values are normalized to the [0, 1] interval; while for logarithmic depth type, the data is further transformed into logarithmic form. This processing step is designed to enhance the usability of depth data and lay the foundation for subsequent depth estimation and analysis. After preprocessing, the generated depth mask is saved in NumPy array format.

[0206] 3D grid map ground truth construction: Since the main purpose of this embodiment is to achieve vehicle detection by constructing a 3D grid map, it is necessary to obtain the ground truth of the 3D grid map. First, the size parameters of the 3D grid map need to be defined, including the size settings of length, width and height, and the size of each unit interval. Then, by reading the CSV file containing vehicle size information, the module extracts the key parameters such as length, width and height of different types of vehicles. Here we use the drawing function of matplotlib to generate the ground truth. The specific steps are: for each height layer, according to the size and position of the vehicle, use a rectangular frame to draw the outline of each vehicle in the figure. After completion, set the range of the coordinate axis and save the intermediate results. Then post-process the generated image, including converting it to a grayscale image, removing the edges and binarizing it to construct it as a 2D occupancy grid map. Finally, the images are spliced at the height layer and saved in NumPy array format, completing the construction of the ground truth of the 3D grid map.

[0207] Dataset collection:

[0208] To meet the needs of vehicle detection tasks, based on the above dataset collection system, through fine scene setting and parameter configuration, ensure that the multi-view, multi-frame data collected has high spatio-temporal consistency and coverage, providing comprehensive support for subsequent model training and verification.

[0209] Parameter setting of collected data

[0210] The dataset collection adopts precise parameter configuration to ensure high-quality collection of each frame of data. The specific parameters are as follows:

[0211] Monocular camera parameters: 120x120 pixels, field of view angle of 60°.

[0212] Array camera structure: a total of 12 monocular cameras are used, arranged in three layers in parallel, with a total field of view angle of about 180°.

[0213] Time interval: the frame rate of data collection is set to 10 frames / second.

[0214] Collection range: the scene of the collection system covers a space of 50m x 100m x 2m, with a grid interval of 0.5m.

[0215] This embodiment has collected 20 different scene data, each with a duration of 7-20s, with a total data size of more than 2000 frames. For the design of traffic scenes, to ensure the diversity of the dataset, the invention collects a variety of different scenes, including:

[0216] Road scene: contains straight roads, curved roads, crossroads and other road conditions, simulating vehicle driving and parking scenes in urban traffic.

[0217] Light environment: Simulate poor light conditions to test the robustness of the vehicle detection system in low light environments such as early morning or dusk.

[0218] Weather conditions: Introduce weather conditions such as rain and fog to test the perception performance of the system in different weather conditions.

[0219] Evaluation index:

[0220] In this embodiment, the intersection over union (IoU) is used as the main evaluation index to measure the detection accuracy of the vehicle foreground area in the three-dimensional grid. IoU calculates the overlap between the predicted area and the true area, which is defined as the ratio of the intersection of the predicted occupancy area and the true occupancy area to the union, expressed as:

[0221]

[0222] where V jl_^ represents the three-dimensional occupancy area predicted by the model, and V an represents the true three-dimensional occupancy area.

[0223] Specifically, in the task of this embodiment, IoU is used to evaluate the detection effect of the vehicle occupancy area in the three-dimensional occupancy grid generated by the model. Since the background area is not the core target in the task, we only calculate the IoU of the vehicle foreground area to measure the accurate prediction of the model on the vehicle occupancy. A higher IoU value indicates that the vehicle occupancy area predicted by the model is highly consistent with the true area, effectively capturing the existence of the vehicle and its position in three-dimensional space.

[0224] Experimental results:

[0225] Under the above evaluation index, the final output result of the model after training on the dataset has an IOU of 0.4613.

[0226] Embodiment three

[0227] This embodiment discloses a vehicle detection system based on a biomimetic compound eye array camera, comprising:

[0228] The feature extraction module is configured to obtain multi-view single camera images, perform feature extraction on multiple single camera images respectively, and generate corresponding camera features and semantic segmentation prediction results;

[0229] The feature generation module is configured to process and map the semantic segmentation prediction results to a three-dimensional grid map through a semantic information-based attention mechanism, and obtain voxel features; and process the voxel features through a depth information-based attention mechanism to generate a final feature representation;

[0230] The multi-camera feature fusion module is configured to fuse the final feature representation of the multi-view by combining 3D convolution and deformable convolution, generate an occupancy of a three-dimensional grid and determine whether a vehicle exists in the three-dimensional grid.

[0231] It should be noted that the feature extraction module, the feature generation module and the multi-camera feature fusion module correspond to the steps in Embodiment One, and the above modules have the same examples and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment One. It should be noted that the above modules can be executed in a computer system such as a set of computer executable instructions as part of the system.

[0232] Embodiment Four

[0233] Embodiment Four of the present application provides an electronic device comprising a memory and a processor, and computer instructions stored on the memory and running on the processor, when the computer instructions are executed by the processor, the steps of the above vehicle detection method based on the bionic compound eye array camera are completed.

[0234] Embodiment Five

[0235] Embodiment Five of the present application provides a computer readable storage medium for storing computer instructions, when the computer instructions are executed by the processor, the steps of the above vehicle detection method based on the bionic compound eye array camera are completed.

[0236] Embodiment Six

[0237] Embodiment Six of the present application provides a computer program product comprising computer programs / instructions, which when executed by a processor, implement the steps of the above vehicle detection method based on the bionic compound eye array camera.

[0238] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The device for performing the functions specified in one flow or multiple flows and / or blocks Figure 1 The device for performing the functions specified in one flow or multiple flows and / or blocks

[0239] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the embodiments herein can be implemented as computer program instructions. Figure 1

[0240] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 The functions of a flow or multiple flows and / or a block or multiple blocks in accordance with the embodiments herein can be implemented as computer program instructions. Figure 1

[0241] The above description of the various embodiments can have emphasized certain aspects of the various embodiments, which description relating to a certain embodiment can be applicable to other embodiments. For the sake of completeness, the description related to one embodiment can be considered to apply to other embodiments described herein, and vice versa.

[0242] The above description is only preferred embodiments of the present application and is not used to limit the present application. The present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.​​

Claims

1. A vehicle detection method based on a bionic compound eye array camera, characterized in that, include: Multiple single-camera images from different perspectives are acquired, and features are extracted from each single-camera image to generate corresponding camera features and semantic segmentation prediction results; wherein, the multiple single-camera images from different perspectives are acquired through a bionic compound eye array camera system. Camera features and semantic segmentation prediction results are processed and mapped to a 3D mesh map using an attention mechanism based on semantic information to obtain voxel features; the voxel features are then processed using an attention mechanism based on depth information to generate the final feature representation. By combining 3D convolution and deformable convolution to fuse the final feature representations from multiple perspectives, the occupancy of the 3D mesh is generated and the presence of vehicles in the 3D mesh is determined. The process of processing camera features and semantic segmentation prediction results using a semantic information-based attention mechanism and mapping them to a 3D mesh map to obtain voxel features specifically includes: The camera features are multiplied element-wise with the corresponding semantic segmentation prediction results to obtain intermediate feature representations; Calculate the foreground proportion of each monocular camera, and determine the attention weight of each monocular camera based on the foreground proportion; obtain the weighted features based on the attention weights and intermediate feature representations; Based on the coordinates and weighted features of each voxel in the 3D mesh map on the image plane, obtain the corresponding voxel features; The process of processing voxel features using a depth-information-based attention mechanism to generate the final feature representation specifically includes: Calculate the baseline depth of the 3D mesh map and the estimated depth value for each individual camera, and determine the attention weights that incorporate depth information; The final feature representation is generated based on attention weights and voxel features.

2. The vehicle detection method based on a bionic compound eye array camera as described in claim 1, characterized in that, Also includes: Obtain the 3D occupied grid sequence; The three-dimensional occupancy grid sequence is processed by a trained temporal feature extraction network to obtain the occupancy probability after temporal fusion and determine whether there is a vehicle.

3. The vehicle detection method based on a bionic compound eye array camera as described in claim 1, characterized in that, The step of extracting features from multiple single-camera images and generating corresponding camera features and semantic segmentation prediction results specifically includes: The Conv-UNext network is used to extract features from multiple single-camera images sequentially to obtain the corresponding camera features. The camera features are then processed sequentially through depthwise separable convolution and the Sigmoid activation function to generate semantic segmentation prediction results.

4. A vehicle detection system based on a biomimetic compound eye array camera, employing the vehicle detection method based on a biomimetic compound eye array camera as described in any one of claims 1-3, characterized in that, include: The feature extraction module is configured to: acquire single-camera images from multiple perspectives, extract features from the multiple single-camera images respectively, and generate corresponding camera features and semantic segmentation prediction results; The feature generation module is configured to: process the semantic segmentation prediction results through an attention mechanism based on semantic information and map them onto a 3D mesh map to obtain voxel features; and process the voxel features through an attention mechanism based on depth information to generate the final feature representation. The multi-camera feature fusion module is configured to fuse the final feature representations from multiple perspectives by combining 3D convolution and deformable convolution, generate the occupancy status of the 3D mesh, and determine whether there are vehicles in the 3D mesh.

5. A biomimetic compound eye array camera system, employing the vehicle detection method based on a biomimetic compound eye array camera as described in any one of claims 1-3, characterized in that, Includes multi-layer SLR cameras and spherical mounts; Multi-layer monocular cameras are arranged on the spherical support along longitudinal spacing, with adjacent layers of monocular cameras arranged in parallel and staggered arrangement; Each layer of monocular cameras includes multiple monocular cameras, which are arranged along a horizontal spacing on the spherical support.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the vehicle detection method based on a bionic compound eye array camera as described in any one of claims 1-3.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the vehicle detection method based on a bionic compound eye array camera as described in any one of claims 1-3.

8. A computer program product comprising a computer program / instructions, characterized in that, When executed by a processor, the computer program / instructions implement the steps of the vehicle detection method based on a bionic compound eye array camera as described in any one of claims 1-3.

Images