An eye array camera-based target detection neural network accelerator

Abstract

The application belongs to the technical field of neural network acceleration control, and discloses a target detection neural network accelerator based on an array camera, which comprises a multi-view image acquisition module, a data preprocessing module, a heterogeneous computing module, a target fusion module and a trajectory generation module. The computing unit of the neural network accelerator is divided into two types of high-precision and low-power consumption. The multi-view images collected by the array camera are evaluated for target value, and high-value images are dynamically allocated to high-precision units, and low-value images are dynamically allocated to low-power consumption units, so that the adaptive scheduling of computing resources is realized, and the excessive calculation of low information density areas can be effectively avoided. At the same time, after the multi-view images are processed heterogeneously, the same target classification is realized, the low-power consumption unit is used to output the detection result as the spatial constraint of the subsequent output of the high-precision unit, cross-path target fusion is realized, and the pipeline blockage and the increase of end-to-end delay are minimized.

Description

Technical Field

[0001] This invention belongs to the field of neural network acceleration control technology, and specifically discloses a target detection neural network accelerator based on a compound eye array camera. Background Technology

[0002] With the continuous increase in urban road traffic flow, the probability of traffic conflicts and the risk of accidents have risen significantly. Implementing high-precision and robust multi-target detection and continuous tracking at road intersections has become a key technical aspect of realizing intelligent traffic management.

[0003] However, with the continuous expansion of urban main roads and multi-lane intersections, traditional monocular or binocular vision systems have limited field of view coverage and insufficient perspective diversity, making it difficult to meet the needs of all-weather, all-scene, and all-view traffic monitoring. Against this backdrop, biomimetic compound eye array cameras, with their structural advantages of wide field of view, multiple perspectives, and high spatiotemporal synchronization, have shown significant potential in the all-domain perception task at complex urban intersections.

[0004] When processing high-concurrency, multi-channel image streams from compound eye array cameras, neural network accelerators become the core hardware support for achieving low-latency inference. However, traditional general-purpose computing architectures face the following drawbacks: First, parallel processing of multi-view images places extremely high demands on computing power, while existing methods generally adopt a uniform and homogenized processing strategy, failing to consider the semantic value differences between different image regions. Because compound eye arrays cover a wide area, some sub-camera views often contain a large amount of static background or distant small targets, whose information density is significantly lower than the high-resolution target area in the central region. Applying the same computational intensity to all data will lead to a resource mismatch problem: insufficient computing power in high-value areas and wasted computing power in low-value areas.

[0005] Secondly, the target detection time varies significantly among different sub-views due to differences in scene complexity. Traditional architectures typically employ a synchronous aggregation mechanism, requiring the slowest path to complete inference before entering the fusion stage. This leads to pipeline congestion and increased end-to-end latency, making it difficult to meet real-time requirements. Summary of the Invention

[0006] In view of this, the present invention aims to propose a target detection neural network accelerator based on a compound eye array camera. By dividing the computing units in the accelerator into high-precision computing units and low-power computing units, a paradigm shift from coarse-grained parallel processing to fine-grained value perception computing is achieved, so as to break through the bottlenecks of existing visual perception systems in terms of efficiency, accuracy and robustness.

[0007] The objective of this invention can be achieved through the following technical solution: a target detection neural network accelerator based on a compound eye array camera, comprising the following modules: a multi-view image acquisition module: using a compound eye array camera set at a traffic intersection to simultaneously acquire multi-view images of traffic participants.

[0008] Data preprocessing module: Adds temporal synchronization identifiers to multi-view images of traffic participants, extracts target bounding boxes through a motion target capture network, and performs target value assessment based on the spatial distribution characteristics of the bounding boxes.

[0009] Heterogeneous computing module: Based on the target value assessment results, multi-view images of traffic participants are assigned to high-precision computing units or low-power computing units for processing.

[0010] Target fusion module: Automatically classifies multi-view images of the same target acquired at the same time, backprojects the target detection results output by the low-power computing unit onto the three-dimensional road coordinate system to form a conical spatial region, and geometrically correlates the accurate detection results output by the high-precision computing unit with the conical spatial region.

[0011] Trajectory generation module: Generates the three-dimensional position of the target based on the target fusion result, and synthesizes the target motion trajectory in time series.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention divides the computing units of the neural network accelerator into two categories: high precision and low power consumption, and constructs a heterogeneous processing path oriented towards visual saliency. By evaluating the target value of multi-view images acquired by the compound eye array camera, high-value images are dynamically allocated to high-precision units and low-value images are allocated to low-power units, realizing adaptive scheduling of computing resources. This can effectively avoid over-computation in low information density areas, achieve on-demand allocation and focused calculation, and effectively improve the energy efficiency ratio and perception efficiency per unit computing power.

[0013] 2. This invention performs heterogeneous processing on multi-view images acquired by a compound eye array camera to classify the same targets. It utilizes the detection results output by the low-power unit as a spatial constraint for the subsequent output of accurate detection results by the high-precision unit, thereby achieving cross-path target fusion. This not only avoids missed detections caused by relying solely on high-precision results, but also fully utilizes the processing delay by guiding the timing coordination of fine matching through coarse positioning, thus minimizing pipeline blockage and end-to-end delay.

[0014] 3. Based on target fusion, this invention generates the three-dimensional position and motion state of the target and constructs a continuous motion trajectory in time series, accurately depicting the dynamic evolution process of traffic participants in three-dimensional road space, effectively improving the continuity and stability of target tracking, and significantly enhancing the perception robustness and behavior prediction ability in complex traffic scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the module data flow in this invention.

[0017] Figure 2 This is a flowchart illustrating the target value assessment process based on the pixel coverage and spatial distribution features of the bounding box in this invention.

[0018] Figure 3 This is a flowchart illustrating the geometric association between the precise detection results output by the high-precision computing unit and the conical spatial region in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention proposes a target detection neural network accelerator based on a compound eye array camera. This accelerator refers to a high-efficiency computing system that integrates a customized hardware architecture and a collaborative optimization algorithm, used to realize multi-view image processing and target detection.

[0021] The entire implementation process includes a multi-view image acquisition module, a data preprocessing module, a heterogeneous computing module, a target fusion module, and a trajectory generation module.

[0022] See Figure 1 As shown, the data flow between modules is as follows: the acquisition module synchronously acquires multi-view images, the preprocessing module performs temporal alignment and preliminary detection, the heterogeneous computing module distributes the data to high-precision or low-power units for processing according to value classification, the target fusion module performs cross-view 3D association, and finally the trajectory generation module outputs a spatiotemporally continuous motion trajectory.

[0023] The functionality of a neural network accelerator relies on the coordinated execution of underlying computing units. By loading pre-defined neural network model weights and instruction streams, the computing units perform convolution, activation, pooling, and other operator operations in a hardware-accelerated manner to complete layer-by-layer feature extraction and target inference of the input image.

[0024] The multi-view image acquisition module is used to simultaneously acquire multi-view images of traffic participants using compound eye array cameras set up at traffic intersections.

[0025] As an application of the above solution, the compound eye array camera consists of multiple sub-camera units, which are distributed in a spatially staggered topological layout to cover the omnidirectional field of view of the road intersection. Through precisely designed installation angles and directional configurations, each sub-camera is responsible for acquiring a local field of view in a specific direction, with appropriate overlap between adjacent fields of view, thereby achieving a panoramic perception of the intersection area with no blind spots and high redundancy.

[0026] The data preprocessing module is used to add time synchronization identifiers to multi-view images of traffic participants, extract target bounding boxes through a motion target capture network, and perform target value assessment based on the pixel coverage and spatial distribution features of the bounding boxes.

[0027] As a preferred implementation of the above scheme, adding a timing synchronization identifier to the multi-view images of traffic participants is performed as follows: a frame synchronization signal generator is deployed at each image sensor interface of the compound eye array camera.

[0028] By connecting the exposure control pins of each image sensor to the main clock source via a synchronous trigger bus, precise synchronization of exposure times is achieved at the electrical level for all imaging channels.

[0029] Add a frame header containing a microsecond-level timestamp to each frame of image data.

[0030] A frame alignment buffer is set in the data transmission channel. When it is detected that all channels have completed the image storage for the current acquisition cycle, a batch data transmission mechanism is triggered to eliminate the data arrival timing deviation between channels.

[0031] A multi-channel data verification unit is set at the input end of the neural network accelerator. By comparing the timestamps of the frame headers of each channel, the timing synchronization is verified, ensuring that the multi-view image data input to the subsequent processing module meets the preset timing synchronization tolerance, and providing reliable time reference support for subsequent spatiotemporal fusion and target association.

[0032] It should be noted that, due to differences in image sensor clock offsets, exposure control, or data transmission paths, the individual sub-cameras in the compound eye array camera may experience microsecond-level time asynchrony in their acquisition times. In subsequent target detection, although the multi-view structure can expand the sensing coverage, strict temporal synchronization of multi-view images is essential for cross-view target association. If there are temporal deviations in the images acquired by each sub-camera, the spatial state of the same physical target will be inconsistent across different viewpoints, leading to problems such as cross-view matching misalignment, increased 3D backprojection errors, trajectory breaks, or increased false alarms. This severely impacts the accuracy of multi-view fusion and the overall robustness of the system.

[0033] As a further preferred implementation of the above scheme, see [link to relevant documentation]. Figure 2 As shown, the target value assessment process is as follows: target capture is performed on each frame of the image based on time-synchronized multi-view images of traffic participants, preliminary target detection results are extracted, and corresponding target bounding boxes are generated.

[0034] In the example of the above operation, the target may, for instance, include traffic participants such as motor vehicles, non-motor vehicles, and pedestrians. Their representation in the image reflects their spatial location, scale, and motion state within the intersection scene. By performing forward inference on each frame of the image using a lightweight object capture network, potential regions of interest can be quickly located, and preliminary bounding boxes can be generated, enabling real-time coarse screening and semantic localization of traffic participants.

[0035] Additionally, the target acquisition step mentioned above primarily aims to determine the presence of predefined detection targets within the image, rather than performing comprehensive processing on the entire image. This step efficiently filters out frames containing targets of interest, thus avoiding unnecessary processing of frames lacking significant information. Once a target is identified, it indicates that the frame contains a subject of interest, and the exact location, size, and relative position within the scene of the target are determined by generating a corresponding target bounding box. This process provides crucial information for subsequent advanced processing and analysis.

[0036] The target coverage ratio is calculated as the ratio of the pixel area covered by the target bounding box in the corresponding image to the total pixel area of ​​the image.

[0037] Understandably, target coverage ratio reflects the relative scale significance of the target in the image's field of view. A higher ratio indicates a larger projected size, higher pixel density, richer texture details and geometric features, and a higher imaging resolution that is more conducive to fine characterization. Therefore, targets with high coverage ratios have stronger feature discriminability in subsequent detection tasks and contribute more to tasks such as classification, localization, and pose estimation.

[0038] Define a rectangular or circular central area of ​​interest based on the image center, and classify the remaining outer areas as non-central areas.

[0039] In one example, the size of the central area of ​​interest can be set to a fixed ratio of the image width and height, such as a 50%×50% area in the center. This range ensures sufficient coverage of the main field of view while avoiding excessive expansion, and can effectively reflect the visual salience of the target in the imaging plane and the reliability of observation.

[0040] The image segmentation method described above is based on the central attention mechanism of human vision, dividing the image plane into a central area of ​​interest (CIFO) and a non-central area. Generally, targets located within the CIFO are more easily perceived with higher priority, resulting in higher image quality and less geometric distortion. If the target's bounding box falls within the CIFO, it indicates that the sub-camera's viewing angle is close to the forward or main observation direction. In this case, the target's pose is highly visible, its feature information is rich, and the acquired texture, contour, and scale information are more discriminative. Conversely, if the target is located in a non-central area, it indicates that the sub-camera's viewing angle is biased towards the side or oblique direction, resulting in significant projection distortion, partial occlusion, or reduced resolution, leading to decreased information integrity and detection reliability.

[0041] For the detected target bounding box, determine whether it falls within the central area of ​​interest.

[0042] As a way to achieve the above scheme, the specific implementation of determining whether it falls within the central area of ​​interest is as follows: extract the closed contour of each target bounding box and the central area of ​​interest, calculate the ratio of the perimeter of the intersection boundary between the two to the perimeter of the complete contour of the target bounding box, and define it as the contour overlap.

[0043] Contour overlap reflects the spatial coverage consistency between the target and the central region. A higher value indicates that the sub-camera captures the target from a more positive angle and has stronger pose visibility. If the target is completely outside or completely inside the central region of interest, the contour overlap is low. When the target is completely outside the central region of interest, it indicates that it is at the edge or in a blind spot, with severe skew, occlusion, or low signal-to-noise ratio. When the target is completely inside the central region of interest, it indicates that the target bounding box size is small, which means that it is far from the camera, has a small projection scale, contains sparse pixel information, and has low semantic feature density and 3D positioning reliability, which is not conducive to high-precision perception.

[0044] Locate the geometric center of each target bounding box and the geometric center of the central interest area, and obtain the Euclidean distance between them, which is denoted as the center offset.

[0045] Center offset is used to quantify the degree of deviation of the target from the principal axis of the image, reflecting the angular offset of the target relative to the camera's main line of sight and the degradation trend of imaging geometry. The larger the offset, the closer the target is to the edge of the field of view, and its projection is more susceptible to lens distortion, resolution reduction, and attitude blur. Conversely, a smaller offset means that the target is located in the high-fidelity imaging area, with better observation conditions, and is suitable for high-confidence perception tasks.

[0046] Compare the overlap between the bounding box of each target and the central area of ​​interest, the center offset, and the judgment limit.

[0047] If the overlap between the bounding box of a target and the central area of ​​interest reaches the overlap limit and the center offset is less than or equal to the allowable offset, then the bounding box of the target is determined to fall within the central area of ​​interest.

[0048] Conversely, if any condition is not met, it is determined that it falls into a non-central region.

[0049] The aforementioned overlap limit can be determined based on statistical analysis. Specifically, a dataset containing various traffic participants is collected in a typical intersection scenario. The overlap distribution of the target bounding box and the central area of ​​interest within the core observation area is statistically analyzed. By analyzing the cumulative distribution function, the minimum value that allows more than 90% of effective targets to be correctly classified is selected as the baseline threshold, which is usually set to 0.6.

[0050] The above allows the offset to be set to 35% to 40% of the image diagonal length. This range corresponds to the low distortion region of the camera's effective imaging, ensuring that the target is located near the principal optical axis and has good attitude observability and three-dimensional positioning accuracy.

[0051] This invention integrates the dual geometric characteristics of spatial coverage relationship and center tendency for a composite criterion for whether a target bounding box falls within the central area of ​​interest. This effectively improves the robustness of the determination in complex scenarios such as large targets crossing regions, partial occlusion, or boundary segmentation, and provides a highly reliable spatial semantic basis for subsequent target value assessment.

[0052] The target coverage percentage of each frame is compared with the coverage threshold. If the target bounding box of a frame is located in the central area of ​​interest and the target coverage percentage reaches the coverage threshold, the frame is marked as a high-value image; otherwise, the frame is marked as a low-value image.

[0053] It should be noted that the above-mentioned target bounding box being located in the central region of interest measures the degree of spatial intersection between the target bounding box and the central region of interest, reflecting whether it is in the main observation field of view. The target coverage ratio measures the relative scale of the target itself in the whole image, reflecting its imaging resolution and detail richness. Even if the target is located in the central region, it may still appear as a large overlap and small area due to the distance. In this case, the target bounding box being located in the central region of interest indicates that the viewpoint is positive and the pose is complete, but the low coverage ratio indicates insufficient pixel density, which is not conducive to high-precision classification or feature extraction.

[0054] Before performing target detection on multi-view images acquired by a compound eye array camera, this invention first performs a target value assessment based on preliminary target recognition. This assessment considers both target coverage percentage and target spatial location, achieving a quantitative classification of information density and perception priority for each frame. This process provides crucial support for subsequent heterogeneous image processing.

[0055] The heterogeneous computing module is used to allocate image data blocks to high-precision computing units or low-power computing units for processing based on the target value assessment results.

[0056] Specifically, high-value images are assigned to high-precision computing units, while low-value images are assigned to low-power computing units.

[0057] The neural network accelerator described in this invention integrates high-precision computing units and low-power computing units in its physical architecture, forming a heterogeneous parallel computing system. The high-precision computing unit consists of hardware circuits optimized for high-throughput, intensive matrix operations, possessing powerful peak computing power and memory bandwidth. It is suitable for performing high-confidence target detection and fine feature extraction on high-value images, ensuring the accuracy of key target identification and positioning. The low-power computing unit is optimized for extreme energy efficiency, employing a lightweight architecture design that supports low-bit operations and sparse activation skipping mechanisms. By simplifying the data path, compressing control logic, and reducing static power consumption, it significantly reduces area and energy consumption while maintaining basic detection capabilities, making it suitable for rapid coarse detection or reduced-resolution inference on low-value images.

[0058] The two types of computing units are deployed collaboratively at the chip level, and resource isolation and dynamic scheduling can be achieved through modern processes.

[0059] The target fusion module is used to automatically classify multi-view images of the same target acquired at the same time, back-project the target detection results output by the low-power computing unit onto the three-dimensional road coordinate system to form a conical spatial region, and geometrically correlate the accurate detection results output by the high-precision computing unit with the conical spatial region.

[0060] In an optional implementation of the present invention, the following operations are automatically categorized: identifying the same target based on the appearance features of the target in multi-view images acquired at the same time.

[0061] Create a unique identifier for each identified identical target and establish a mapping table between the target identifier and the corresponding multi-view image.

[0062] This invention achieves semantic alignment and identity consistency association across views by automatically classifying the same target in multi-view images acquired at the same time. Its core function lies in the fact that the same physical target often appears in the overlapping fields of view of multiple sub-cameras. Due to differences in viewpoints, it may be classified as a high-value image in some views and a low-value image in others. To ensure the integrity and consistency of target representation, it is necessary to fuse multi-granularity feature outputs from heterogeneous computing pathways. In this process, target classification establishes target correspondences across views and processing paths, providing crucial cross-unit association indexes for subsequent multi-source information fusion. This ensures that the fine features output by high-precision units and the supplementary observations provided by low-power units can be accurately aligned to the same target entity, thereby supporting fusion reasoning based on geometric and semantic constraints and improving the accuracy and robustness of the overall perception system.

[0063] In further optional embodiments of the present invention, see [link to relevant documentation]. Figure 3 As shown, the geometric correlation includes the following processing steps: reading the intrinsic and extrinsic parameter matrices of each sub-camera of the compound eye array camera from the camera parameter memory.

[0064] In an example of the above operations, the intrinsic parameter matrix contains focal length, principal point coordinates, distortion coefficients, etc., while the extrinsic parameter matrix is ​​used to describe the rotation and translation transformations of each sub-camera relative to a unified world coordinate system or road coordinate system.

[0065] These parameters collectively constitute the core of the multi-view geometric calibration model, reflecting the precise pose distribution and projection characteristics of each sub-camera in space. They are the foundation for achieving accurate mapping between pixel coordinates and real-world coordinates. Their calibration accuracy directly affects the accuracy of inverse perspective mapping, 3D backprojection, and cross-view target association, and is a crucial prerequisite for ensuring the geometric consistency of the multi-view fusion system.

[0066] The coordinates of the two-dimensional detection box output by the low-power computing unit are combined with the intrinsic and extrinsic parameters of the corresponding sub-camera to perform inverse perspective mapping to the three-dimensional road coordinate system, generating a cone-shaped spatial region.

[0067] As a specific embodiment of the above scheme, the four corner points of the detection frame are solved by the camera projection equation into spatial ray beams that start from the image point and extend along the optical center. Assuming that the target is located on the ground plane or within a certain height range, the intersection of these rays with the three-dimensional space forms a cone-shaped search area with directionality and spatial expansion.

[0068] Because low-power computing units typically employ lightweight network structures and reduced-resolution inputs, the output detection boxes exhibit scale and positional uncertainties. The back-projection result is not a single point, but rather a cone-shaped spatial region extending in three-dimensional space, representing the possible physical spatial range of the target.

[0069] The precise detection frame output by the high-precision computing unit is also projected onto the same three-dimensional road coordinate system based on the intrinsic and extrinsic parameter information of its sub-camera to obtain the high-precision three-dimensional position of the target.

[0070] Because the high-precision computing unit uses high-resolution input and a complex model architecture, the output detection box can be accurately positioned by projection.

[0071] The geometric inclusion relationship between the high-precision three-dimensional position of the target and the conical spatial region generated by the low-power computing unit is identified. If the position falls within the conical spatial region, it is determined that the two correspond to the same physical target.

[0072] When the target is identified as the same physical target, the system outputs a fused, high-precision, and highly complete 3D target state.

[0073] Considering that the low-power path quickly outputs a coarse position, while the high-precision path outputs precise 3D coordinates with a delay, if both correspond to the same physical entity, then according to the principle of multi-view geometric consistency, the high-precision 3D coordinates should be located within the conical spatial region generated by the low-power path. This criterion constructs a cross-path correlation verification mechanism based on geometric constraints, possessing good physical interpretability and error tolerance. This mechanism not only achieves effective fusion of the output results from heterogeneous computing units, reflecting the integrity of the target in spatial coverage and state representation, but also fully utilizes the processing delay of the high-precision path. While waiting for the refined results, it establishes spatial constraints using the prior information of the low-power path, achieving pipeline optimization that trades time for collaboration, significantly improving the system's spatiotemporal alignment efficiency and resource utilization efficiency in multi-view perception tasks.

[0074] As an example of the application scenario of the above scheme, it is shown below: Time T0: The compound eye array camera synchronously acquires sub-image A located on the road in front and sub-image B located on the side curb.

[0075] At time T1: The low-power computing unit quickly processes subgraph B and finds a coarse vehicle detection box. The system immediately backprojects it into a huge three-dimensional cone-shaped space region and creates a target to be confirmed in the global list.

[0076] Time T2: The high-precision computing unit has finished processing sub-image A, accurately detected a car and calculated its three-dimensional coordinates.

[0077] Time T3: The fusion center detects that the three-dimensional coordinates of the vehicle are located within the cone-shaped spatial region of the target to be confirmed.

[0078] Result: The system confirmed that the vehicles in subgraph A and subgraph B are the same vehicle, and generated a fused target with a precise 3D location and high confidence.

[0079] In an innovative implementation of the above scheme, when a single conical spatial region contains multiple precise three-dimensional positions, the vertical distance between each candidate three-dimensional target position point and the effective central axis of the conical spatial region is calculated. This axis is defined as a ray that originates from the optical center of the corresponding sub-camera, extends along the main line of sight, and is projected through the center of the bounding box.

[0080] The candidate 3D target location point with the smallest vertical distance is selected as the matching point.

[0081] It should be noted that when multiple candidate 3D positions from high-precision pathways exist within a single conical spatial region, this phenomenon stems from the heterogeneous allocation of compound eye array images to low-power and high-precision units for parallel processing. This leads to multiple responses for the same target from different viewpoints, or introduces mismatches of neighboring targets. In this case, a nearest-spatial-distance matching strategy is adopted: the vertical distance from each candidate point to the central axis of the conical region is calculated, and the smallest distance is selected as the optimal associated target. This distance reflects the spatial consistency between the candidate position and the geometric priors provided by the low-power pathways, aiming to identify the corresponding entity that best conforms to the multi-view geometric constraints from redundant or interfering terms, effectively suppressing mismatches and improving the accuracy and robustness of cross-path target matching.

[0082] Spatial association confidence is assessed based on the vertical distance between the matching point and the effective central axis of the conical spatial region. The specific assessment formula is as follows: In the formula This represents the vertical distance between the matching point and the effective central axis of the conical spatial region. This represents the scale normalization coefficient, used to adjust the penalty strength of distance error on confidence, ensuring that the output value range is in the (0, 1] interval, reflecting the reliability of the matching results in three-dimensional geometric consistency.

[0083] The exponential decay form, used as the theoretical basis for the aforementioned spatial correlation confidence assessment, stems from a classic assumption in sensor fusion and state estimation: observation errors typically follow a zero-mean Gaussian distribution. Under this premise, this form can be interpreted as the posterior likelihood of a candidate match given a geometrical bias; that is, the closer the matching point is to the central axis of the conical spatial region, the higher the probability that it belongs to the same physical target as the output of the low-power path. This modeling approach has a rigorous statistical foundation and conforms to the optimal estimation criterion in multi-source sensing fusion.

[0084] Furthermore, compared to the linear decay caused by directly using vertical distance, introducing a squared vertical distance term can construct a non-linear penalty mechanism: the confidence level decreases slowly in the small error range, reflecting tolerance for reasonable positioning deviations; while in the large error range, it rapidly approaches zero, achieving strong suppression of significant deviations. This characteristic effectively strengthens the core geometric constraint of line-of-sight alignment, ensuring that high confidence can only be obtained when candidate points are highly consistent along the main line-of-sight direction in three-dimensional space, thereby improving the accuracy and robustness of cross-path target association.

[0085] The spatial correlation confidence of the matching points is compared with the confidence threshold. The confidence threshold can be calculated by collecting a large amount of known true multi-view data in typical traffic scenarios, calculating the spatial correlation confidence value of each matching pair, and marking whether it is a correct match. By plotting ROC curves or PR curves, the value that maximizes the F1-score or reduces the false match rate below the tolerance level, such as 5%, is selected as the confidence threshold. When the spatial correlation confidence of the matching point is lower than the confidence threshold, the re-detection and target fusion verification process of the target in the high-precision computing unit and the low-power computing unit is initiated.

[0086] It's important to note that when a matching point fails to match successfully during the re-detection and target fusion verification process in both the high-precision and low-power computing units, this typically indicates insufficient confidence in the current optimal matching point to ensure its accuracy and reliability. In this case, the default matching assumption may contain significant errors or be completely inaccurate, requiring further validation of other potential matching candidates to confirm the target's exact location. The system then shifts to evaluating the second-closest candidate 3D location in terms of vertical distance for spatial correlation confidence assessment. This step aims to improve the overall accuracy and robustness of the tracking system by exploring suboptimal but potentially more reliable matching options.

[0087] The trajectory generation module is used to generate the three-dimensional position of the target based on the target fusion result and synthesize the target motion trajectory in time series.

[0088] Specifically, the target motion trajectory is generated as follows: within a continuous time window, the three-dimensional state information of the same target at different timestamps is collected and output by the high-precision computing unit and the low-power computing unit after spatial fusion. The three-dimensional state includes the target's three-dimensional position and motion attitude in the global road coordinate system.

[0089] The target motion trajectory is generated by trajectory smoothing filtering, which is a discrete three-dimensional position sequence of the target after spatial fusion.

[0090] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0091] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0092] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0094] Finally, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A target detection neural network accelerator based on a compound eye array camera, characterized in that, Includes the following modules: Multi-view image acquisition module: Simultaneously acquires multi-view images of traffic participants using compound eye array cameras set up at traffic intersections; Data preprocessing module: Adds temporal synchronization identifiers to multi-view images of traffic participants, extracts target bounding boxes through a moving target capture network, and performs target value assessment based on the pixel coverage and spatial distribution features of the bounding boxes. Based on time-synchronized multi-view images of traffic participants, target capture is performed on each frame of the image, preliminary target detection results are extracted, and corresponding target bounding boxes are generated. The target coverage ratio is calculated as the ratio of the pixel area covered by the target bounding box in the corresponding image to the total pixel area of ​​the image. Define a rectangular or circular central interest area based on the image center, and classify the remaining outer parts as non-central areas; For the detected target bounding box, determine whether it falls within the central area of ​​interest; The target coverage ratio of each frame image is compared with the coverage threshold. If the target bounding box of a frame image is located in the central interest area and the target coverage ratio reaches the coverage threshold, the frame image is marked as a high-value image; otherwise, the frame image is marked as a low-value image. Heterogeneous computing module: Based on the target value assessment results, multi-view images of traffic participants are assigned to high-precision computing units or low-power computing units for processing; Target fusion module: Automatically classifies multi-view images of the same target acquired at the same time, back-projects the target detection results output by the low-power computing unit onto the three-dimensional road coordinate system to form a cone-shaped spatial region, and geometrically correlates the accurate detection results output by the high-precision computing unit with the cone-shaped spatial region; Trajectory generation module: Generates the three-dimensional position of the target based on the target fusion result, and synthesizes the target motion trajectory in time series.

2. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The process of adding a time synchronization identifier to the multi-view images of traffic participants is as follows: Deploy a frame synchronization signal generator at each image sensor interface of the compound eye array camera; The exposure control pins of each image sensor are connected to the main control clock source via a synchronous trigger bus; Add a frame header containing a microsecond-level timestamp to each frame of image data; A frame alignment buffer is set in the data transmission channel. When it is detected that all channels have completed the image storage for the current acquisition cycle, the batch data transmission mechanism is triggered. A multi-channel data verification unit is set at the input of the neural network accelerator to verify timing synchronization by comparing the timestamps of the frame headers of each channel.

3. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The process for determining whether each detected target bounding box falls within the central area of ​​interest is as follows: Extract the closed contour of each target bounding box and the central region of interest, and calculate the ratio of the perimeter of the intersection boundary between the two to the perimeter of the complete contour of the target bounding box, which is defined as the contour overlap. Locate the geometric center of each target bounding box and the geometric center of the central interest area, and obtain the Euclidean distance between them, which is denoted as the center offset. Compare the overlap between the bounding box of each target and the central area of ​​interest, as well as the center offset, with the judgment limit: If the overlap between the bounding box of a target and the central area of ​​interest reaches the overlap limit and the center offset is less than or equal to the allowable offset, then the bounding box of the target falls within the central area of ​​interest. Conversely, if any condition is not met, it is determined that it falls into a non-central region.

4. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The heterogeneous computing module is implemented as follows: High-value images are assigned to high-precision computing units, while low-value images are assigned to low-power computing units.

5. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The automatic classification of multi-view images of the same target acquired at the same time is described below: Identify the same target based on its appearance features in multi-view images acquired at the same time; Create a unique identifier for each identified identical target and establish a mapping table between the target identifier and the corresponding multi-view image.

6. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The geometric association includes the following processing steps: Read the intrinsic and extrinsic parameter matrices of each sub-camera of the compound eye array camera from the camera parameter memory; The coordinates of the two-dimensional detection box output by the low-power computing unit are combined with the intrinsic and extrinsic parameters of the corresponding sub-camera to perform inverse perspective mapping to the three-dimensional road coordinate system, generating a cone-shaped spatial region. The high-precision detection frame output by the high-precision computing unit is projected onto the same three-dimensional road coordinate system to obtain the high-precision three-dimensional position of the target; The geometric inclusion relationship between the high-precision three-dimensional position of the target and the conical spatial region generated by the low-power computing unit is identified. If the position falls within the conical spatial region, it is determined that the two correspond to the same physical target. When the targets are identified as the same physical target, output the fused 3D target state.

7. The target detection neural network accelerator based on a compound eye array camera as described in claim 6, characterized in that: The geometric association also includes the following: When a single conical spatial region contains multiple precise 3D locations, calculate the vertical distance between each candidate 3D target location point and the effective central axis of the conical spatial region. This axis is defined as a ray that originates from the optical center of the corresponding sub-camera, extends along the main line of sight, and is projected through the center of the bounding box. The candidate 3D target location point with the smallest vertical distance is selected as the matching point; Spatial association confidence is assessed based on the vertical distance between the matching point and the effective central axis of the conical spatial region. The specific assessment formula is as follows: In the formula This represents the vertical distance between the matching point and the effective central axis of the conical spatial region. Represents the scale normalization coefficient; The spatial correlation confidence of the matching point is compared with the confidence threshold. When the spatial correlation confidence of the matching point is lower than the confidence threshold, the re-detection and target fusion verification process of the target in the high-precision computing unit and the low-power computing unit is initiated.

8. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The trajectory generation module includes the following processing steps: Within a continuous time window, the three-dimensional state information of the same target at different timestamps is collected and output by a high-precision computing unit and a low-power computing unit after spatial fusion. The three-dimensional state includes the target's three-dimensional position and motion attitude in the global road coordinate system. The target motion trajectory is generated by trajectory smoothing filtering, which is a discrete three-dimensional position sequence of the target after spatial fusion.

9. The target detection neural network accelerator based on a compound eye array camera as described in claim 1, characterized in that: The compound eye array camera is configured as follows: The compound eye array camera consists of multiple sub-camera units, which are distributed in a spatially staggered topological layout to cover the omnidirectional field of view of the road intersection.

Images