3D target detection realization method and system based on intersection multi-camera cooperation

Abstract

The invention relates to the technical field of intelligent network connection automobiles and intelligent traffic, in particular to a 3D target detection implementation method and system based on intersection multi-camera cooperation. The multiple cameras are synchronized through GNSS time service; establishing a unified ENU coordinate system and calibrating internal and external parameters of the camera; collecting multi-camera image data; 3D target detection is carried out by using an improved ImVoxelNet network; performing multiple times of matching based on positions, categories and IOU on the plurality of 3D target detection frame sequences; and outputting a unified sensing result of the whole intersection. According to the method, through combination of multi-camera cooperation and 3D target detection, the target detection precision and the positioning accuracy are remarkably improved, and the problems of shielding and truncation in a complex traffic scene are effectively solved. The system can obtain the category, geometric dimension, accurate position and orientation angle information of the target at the same time, and provides comprehensive and reliable sensing data for an intelligent traffic system.

Description

Technical Field

[0001] The present invention relates to the field of intelligent networked vehicles and smart transportation technology, and in particular to a method and system for realizing 3D target detection based on the collaboration of multiple cameras at an intersection. Background Art

[0002] With the rapid development of technologies such as the Internet of Things, artificial intelligence, and 5G, intelligent transportation systems have emerged to alleviate road congestion and reduce traffic accidents. Intelligent transportation systems apply computer technology, information technology, data communication technology, sensor technology, and artificial intelligence to transportation management, taking vehicles, roads, people, and the environment as comprehensive considerations to create an integrated transportation system. At present, most of the research on intelligent transportation systems focuses on vehicle-mounted equipment. By installing sensors such as millimeter-wave radar, lidar, and cameras on the car, the surrounding environment information is collected and the surrounding environment is perceived. However, due to the limited height of the vehicle and the limited perception range of the on-board sensor, the vehicle-side perception effect is not good under complex road conditions. In order to make up for the shortcomings of the vehicle-side intelligent system, the vehicle-road cooperative intelligent transportation system was introduced. Based on wireless communication and the new generation of Internet technology, the vehicle-road cooperative system deploys various sensors on the road side to perceive the traffic environment. Through the vehicle network communication technology, the traffic condition information obtained is sent to nearby vehicles, expanding the perception range of the intelligent networked vehicle, ensuring traffic safety and improving traffic efficiency.

[0003] The current roadside visual perception is mainly based on 2D target perception. The image is processed by a convolutional neural network to obtain a sequence of 2D target detection frames. According to the camera intrinsic parameter matrix and extrinsic parameter matrix calculated in advance, the pixel coordinate system coordinates are converted into spatial coordinate system coordinates. However, this conversion is linear. As the target moves away from the camera, the position error obtained after the conversion will gradually increase, and it is impossible to provide accurate position information of the target. In addition, 2D target perception cannot obtain the 3D size information and orientation angle information of each target, but this information is particularly important for the path planning and obstacle avoidance of intelligent connected vehicles. Therefore, 3D target detection in images has gradually attracted widespread attention. In addition, most existing roadside camera perceptions only detect single camera image data, without considering the fusion of multi-point and multi-view image data. In actual complex traffic scenes, it is often necessary to fuse multi-point and multi-view camera image data for continuous target detection and tracking, so as to improve the accuracy of target detection and expand the perception range. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention is proposed.

[0005] Therefore, the problem to be solved by the present invention is how to use the current roadside visual perception as the main 2D target perception. The image is processed by a convolutional neural network to obtain a 2D target detection frame sequence. According to the camera intrinsic parameter matrix and extrinsic parameter matrix calculated in advance, the pixel coordinate system coordinates are converted into spatial coordinate system coordinates. However, this conversion is linear. As the target moves away from the camera, the position error obtained after the conversion will gradually increase, and the accurate position information of the target cannot be provided. In addition, 2D target perception cannot obtain the 3D size information and orientation angle information of each target, but this information is particularly important for the path planning and obstacle avoidance of intelligent networked vehicles. Therefore, 3D target detection of images has gradually attracted widespread attention. In addition, most existing roadside camera perceptions only detect single camera image data, without considering the fusion of multi-point and multi-view image data. In actual complex traffic scenes, it is often necessary to fuse multi-point and multi-view camera image data for continuous target detection and tracking, thereby improving target detection accuracy and expanding the perception range.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, an embodiment of the present invention provides a method for realizing 3D target detection based on the cooperation of multiple cameras at an intersection, which includes: performing time synchronization of multiple cameras at an intersection through GNSS timing, and collecting image data of multiple cameras;

[0008] Establish a unified spatial coordinate system, calibrate the internal and external parameters of the camera, convert each camera's coordinate system into a unified ENU coordinate system, and realize the spatial alignment of multiple cameras at the intersection;

[0009] Perform 3D object detection on the collected intersection multi-camera images and output 3D object detection frames;

[0010] Combined with the internal and external reference relationship of multiple cameras at the intersection, the sequence of 3D object detection frames is matched based on position, category, IOU and re-matched;

[0011] The entire intersection perception result is output based on the matching results to achieve collaborative output between multiple cameras. The results include target category, geometric size, location information and orientation angle.

[0012] As a preferred solution of the 3D target detection implementation method based on the cooperation of multiple cameras at an intersection of the present invention, wherein: the time synchronization of multiple cameras at the intersection through GNSS timing includes synchronizing the cameras using GNSS satellite timing to ensure that the timestamps of data collected by different cameras at the same time are synchronized;

[0013] Establish a unified ENU coordinate system, represent the same intersection area, select the ENU coordinate system as the reference coordinate system of multiple cameras, after calibration, the camera data can be directly converted to the ENU coordinate system, thereby achieving multi-camera space unification;

[0014] Perform 3D target detection on the collected multiple image data and output the 3D target detection frame, including using the improved target detection network ImVoxelNet to perform multi-category 3D target detection on the image to obtain the 3D target detection frame.

[0015] As a preferred solution of the 3D target detection implementation method based on the cooperation of multiple cameras at intersections described in the present invention, wherein: the multiple cameras at intersections are roadside cameras, and the 3D target detection method represents the roadside cooperative perception of multiple cameras, multiple points, and multiple perspectives;

[0016] The roadside camera is a high-definition camera;

[0017] The multi-point position refers to multiple cameras installed and deployed at different locations of the intersection;

[0018] Deploy multiple sets of intelligent roadside poles in the same intersection area, wherein the intelligent roadside poles have cameras, edge computing units, and roadside communication units;

[0019] The multi-view refers to that multiple cameras deployed at different points or the same point have different perception ranges and perception perspectives, so as to expand the range of roadside perception.

[0020] As a preferred solution of the 3D target detection implementation method based on the cooperation of multiple cameras at intersections described in the present invention, wherein: the synchronization of the data timestamps collected by different cameras at the same time includes real-time transmission of the original video images of multiple cameras at the same intersection to the edge computing unit;

[0021] The calibration represents that after the roadside camera is installed, the internal parameters of the cameras on different roadside poles are calibrated using the checkerboard calibration method;

[0022] External parameter calibration is performed on different cameras respectively to obtain external parameter calibration matrices from multiple sets of camera coordinate systems to the ENU coordinate system, thereby achieving spatial alignment of cameras on different roadside poles.

[0023] As a preferred solution of the 3D target detection implementation method based on intersection multi-camera collaboration of the present invention, the multi-category 3D target detection of images using the improved target detection network ImVoxelNet can be performed by the following steps:

[0024] Based on the original ImVoxelNet target detection network model, the model input image size is adjusted to achieve super-resolution detection and improve the accuracy of small target detection;

[0025] Modify ImVoxelNet to meet multi-category requirements, modify the anchor settings, set the anchor size based on the average size of the category, and better extract the features of each category;

[0026] Perform 3D convolution to enhance the model's 3D feature representation performance;

[0027] Finally, an inference model in ONNX format is generated, and the TensorRT framework is used to accelerate model inference and output 3D object detection sequences of multiple images.

[0028] As a preferred solution of the 3D target detection implementation method based on the cooperation of multiple cameras at intersections described in the present invention, the internal and external reference relationship of the multiple cameras at intersections is combined by the following steps:

[0029] Combined with the internal and external reference relationship of multiple roadside cameras, the sequences of multiple 3D target detection frames are matched based on position, category, IOU and re-matched, including the following steps:

[0030] Select two sequences from multiple 3D object detection box sequences, namely sequence a1 and sequence a2, and initialize the result sequence b;

[0031] The matching matrix M constructed from sequences a1 and a2 p And the final matching matrix M f , combined with the internal and external parameters of multi-camera calibration, the positions and orientation angles of the 3D object detection frames of the two sequences are converted from the camera coordinate system to the ENU coordinate system. If the center point of the 3D object detection frame of the a1 sequence falls within the 3D object detection frame of the a2 sequence, and the target classification results of the two are consistent, then the two 3D frames are preliminarily screened to meet the matching conditions; that is, if the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j If the matching condition is met, then M p (i, j) = 1, otherwise M p (i, j) = 0;

[0032] Convert the 3D object detection frames of the two sequences to the ENU coordinate system and calculate the intersection over union (IOU) between the 3D object detection frames of the two sequences, i.e., the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j The intersection-over-union ratio is expressed as IOU(i, j);

[0033] Traverse all 3D target detection boxes in the a1 sequence. If the i-th 3D box bbox3d i There is only one pre-matched 3D box, that is, only M p (i, j) = 1, then M f (i, j) = 1;

[0034] If M p (i, j) = 1 and M p If IOU(i, k)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(i, k), then M f (i, j) = 1, otherwise M f (i, k) = 1;

[0035] If M p (i, j) = 1 and the jth 3D box bbox3d j Already with the mth 3D box bbox3d of the a1 sequence m Match, that is, M p If Mf(m, j)=1 and Mf(i, j)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(m, j), then Mf(m, j)=1. f (i, j) = 1, the mth 3D box bbox3d of the a1 sequence m Repeat this step, otherwise M f (m, j) = 1, the i-th 3D box bbox3d of the a1 sequence i Repeat this step again;

[0036] Get the final matching matrix M f ;

[0037] According to the matching matrix M f , if M f m, n) = 1, then extract the mth 3D box bbox3d of the a1 sequence m and the nth 3D box bbox3d of the a2 sequence n The size information, position information, and orientation angle information of the target are weighted and added according to the detection confidence of the target, and the weighted target is incorporated into the result sequence b. If there is an unmatched target between the a1 sequence and the a2 sequence, the target is directly incorporated into the sequence b;

[0038] If the number of 3D object detection sequences is more than 2, then take a sequence a3 that has not been matched, and repeat the above "constructing the matching matrix M based on sequences a1 and a2" for sequence a3 and sequence b. p And the final matching matrix M f -According to the matching matrix M f , if M f(m, n) = 1" step, until there is no unmatched target sequence, and the final 3D target detection result sequence b is obtained.

[0039] As a preferred solution of the 3D target detection implementation method based on the cooperation of multiple cameras at an intersection of the present invention, wherein: the entire intersection perception result is output according to the matching result to achieve the cooperative output between multiple cameras, and the result includes the target category, geometric size, location information and orientation angle;

[0040] The final merged 3D object detection result sequence is used as the unified perception result of the entire intersection, that is, the output of the collaborative work between multiple cameras at the intersection.

[0041] In a second aspect, an embodiment of the present invention provides a 3D target detection implementation system based on the cooperation of multiple cameras at an intersection, which includes a collection module, wherein multiple cameras at the intersection are synchronized by GNSS timing to collect image data of multiple cameras;

[0042] Construct the module, establish a unified spatial coordinate system, calibrate the internal and external parameters of the camera, convert each camera coordinate system into a unified ENU coordinate system, and realize the spatial alignment of multiple cameras at the intersection;

[0043] The detection module performs 3D object detection on the collected intersection multi-camera images and outputs the 3D object detection frame;

[0044] The matching module combines the internal and external reference relationships of multiple cameras at the intersection to match the sequence of 3D object detection frames based on position, category, IOU, and IOU.

[0045] The output module outputs the entire intersection perception result based on the matching results, realizing the coordinated output between multiple cameras. The results include target category, geometric size, location information and orientation angle.

[0046] In a third aspect, an embodiment of the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program instructions are executed by the processor, the steps of the method for implementing 3D target detection based on intersection multi-camera collaboration as described in the first aspect of the present invention are implemented.

[0047] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program instructions are executed by a processor, the steps of the method for implementing 3D target detection based on intersection multi-camera collaboration as described in the first aspect of the present invention are implemented.

[0048] The beneficial effects of the present invention are as follows: for the perception features of multiple cameras at the same intersection, a 3D target detection solution based on the coordination of multiple cameras at the intersection is proposed. 3D target detection is performed on images to solve the problem that the previous 2D target detection cannot accurately locate the target and cannot obtain the target size information and orientation angle information. It can realize intelligent processing of the perception data of multiple cameras on the road side, solve the problem that the performance of a single camera in detecting blocked targets and truncated targets is poor, greatly improve the accuracy of roadside scene target detection, expand the perception range, and have high detection efficiency and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a flow chart of a method for realizing 3D target detection based on multi-camera collaboration at intersections;

[0051] Figure 2 A diagram of a computer device for implementing a 3D target detection method based on the cooperation of multiple cameras at an intersection;

[0052] Figure 3 It is an overall schematic diagram of the implementation method of 3D target detection based on multi-camera collaboration at intersections;

[0053] Figure 4 This is a deployment diagram of the 3D target detection implementation method based on multi-camera collaboration at intersections. DETAILED DESCRIPTION

[0054] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0056] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0057] Example 1

[0058] Reference Figure 1-2 , which is the first embodiment of the present invention, provides a method for implementing 3D target detection based on the collaboration of multiple cameras at intersections, which is suitable for roadside perception in traffic scenes such as parks, cities, towns, and highways, and is particularly suitable for roadside collaborative perception with multiple cameras, multiple points, and multiple perspectives. The multi-point position is characterized in that multiple sets of intelligent roadside poles equipped with high-definition cameras, edge computing units, roadside communication units, GPS positioning modules and other equipment are deployed in the same intersection area; the multi-perspective is characterized in that multiple cameras deployed at different points or the same point have different perception ranges and perception perspectives, respectively, so as to expand the scope of roadside perception as much as possible. Figure 3 As shown, the most common intersection scene is a crossroad, where four roads intersect, each road is equipped with an intelligent roadside pole, and the roadside pole is equipped with a high-definition camera, and the four cameras jointly shoot the same intersection from four different perspectives. The method specifically includes the following steps:

[0059] S100: Multiple cameras at the intersection are synchronized through GNSS timing to collect image data from multiple cameras;

[0060] Preferably, Figure 4 As shown, GNSS receivers are installed on four smart roadside poles, which receive precise time signals from GNSS satellites. Through adapters, the cameras and GNSS are connected so that the internal clocks of the cameras are synchronized with the GNSS time, ensuring that the data recorded by all cameras has a unified and accurate timestamp. The camera data is first transmitted to the mobile edge computing units, which initially process the data and perform timestamp verification, video analysis, and data aggregation. The mobile edge computing units check the timestamps of the data from each camera to ensure that all data is recorded and transmitted under the same time standard. If time deviation is detected, the relevant settings can be adjusted or the time synchronization can be re-performed.

[0061] Collect image data from multiple cameras at intersections for subsequent target detection. It is necessary to collect image data from four cameras at an intersection. Send the image data to the edge computing unit for target detection.

[0062] S101: Time synchronization of multiple cameras at the intersection through GNSS timing includes synchronizing the cameras using GNSS satellite timing to ensure that the timestamps of data collected by different cameras at the same time are synchronized;

[0063] S200: Establish a unified spatial coordinate system, perform camera internal and external parameter calibration, convert each camera coordinate system into a unified ENU coordinate system, and realize spatial alignment of multiple cameras at the intersection;

[0064] S201: Establishing a unified ENU coordinate system, indicating that in the same intersection area, the ENU coordinate system is selected as the reference coordinate system of multiple cameras. After calibration, the camera data can be directly converted to the ENU coordinate system, thereby achieving multi-camera spatial unification;

[0065] Preferably, the step of establishing a unified ENU coordinate system for multiple cameras on the roadside includes: in the same intersection area, selecting the ENU coordinate system as the reference coordinate system for multiple cameras, and after calibration, the camera data can be directly converted to the ENU coordinate system, thereby realizing a unified coordinate system for multiple camera data.

[0066] The steps for calibrating multiple camera sensors include: after the roadside cameras are installed, the intrinsic parameters of the cameras on different roadside poles are calibrated using the checkerboard calibration method; the extrinsic parameters of different cameras are calibrated separately to obtain the extrinsic parameter calibration matrix from multiple sets of camera coordinate systems to the ENU coordinate system, thereby achieving spatial alignment of the cameras on different roadside poles.

[0067] The four cameras (camera 1, camera 2, camera 3, and camera 4) at the intersection were calibrated. The chessboard calibration method was used to calibrate the internal parameters of the four cameras. The chessboard was placed in front of different orientations and angles, and multiple sets of images were taken for each camera. Usually, in order to improve the accuracy and robustness of the calibration, each camera needs to capture at least 10-20 chessboard images at different angles. The captured images were analyzed using computer vision software to automatically detect the inner corner points of the chessboard in each image. The precise position of the corner points will be used for subsequent camera internal parameter calculations. Using the detected corner point coordinates, combined with the actual physical size of the chessboard, a camera calibration algorithm (such as Zhang Zhengyou calibration method) is used to calculate the internal parameters of each camera to obtain the internal parameter matrix. The calculation results of the internal parameters are optimized by comparing the reprojection errors of different images. The reprojection error refers to the error between the original corner point and the corner point reprojected back to the image plane using the calculated internal parameters and distortion parameters. The smaller the error, the more reliable the calibration result.

[0068] Prepare a test car (intelligent connected car), install RTK on the car, and obtain the position of the test car in the ENU coordinate system. Match the position of the camera coordinate system of each camera shooting the test car with the position of the ENU coordinate system, use PnP calculation to obtain the external parameter matrix of each camera, and iterate the optimization algorithm to reduce the error and improve the accuracy of the overall system. Through the internal parameter matrix and the external parameter matrix, the data of different cameras can be converted to a unified ENU coordinate system.

[0069] S202: The multiple cameras at the intersection are roadside cameras, and the 3D object detection method represents the roadside collaborative perception of multiple cameras, multiple points, and multiple perspectives;

[0070] The roadside camera is a high-definition camera;

[0071] Multi-point means that multiple cameras are installed and deployed at different locations of the intersection;

[0072] Deploy multiple sets of smart roadside poles in the same intersection area. The smart roadside poles are equipped with cameras, edge computing units, and roadside communication units.

[0073] Multi-view refers to multiple cameras deployed at different points or the same point with different perception ranges and perception perspectives to expand the range of roadside perception.

[0074] S203: ensuring that the timestamps of data collected by different cameras at the same time are synchronized, including real-time transmission of original video images of multiple cameras at the same intersection to the edge computing unit;

[0075] After the roadside cameras are installed, the checkerboard calibration method is used to calibrate the internal parameters of the cameras on different roadside poles.

[0076] External parameter calibration is performed on different cameras respectively to obtain external parameter calibration matrices from multiple sets of camera coordinate systems to the ENU coordinate system, thereby achieving spatial alignment of cameras on different roadside poles.

[0077] S300: Perform 3D object detection on the collected intersection multi-camera images and output a 3D object detection frame;

[0078] Preferably, the improved target detection network ImVoxelNet is used to perform multi-category 3D target detection on the image to obtain a 3D target detection frame. The model input image size is adjusted to achieve super-resolution detection and improve the accuracy of small target detection. The input image size here is 1024*512. The backbone layer of the network uses ResNet-50 to extract image features, and the image features are aggregated using a feature pyramid network structure. The aggregated image features are dimensionally upgraded using camera parameters (intrinsic parameters and external parameters) to convert them from 2D features to 3D voxel features. The 3D voxel features are processed using a 3D convolution structure. The 3D anchor setting is modified to set the size of the anchor based on the average size of the category, so as to better extract the features of each category and enhance the multi-category perception performance. Finally, a flattening process is added to convert the 3D voxel features into 2D BEV features. The 2D BEV features are subjected to 2D convolution and regression operations to obtain information on the position, size, and orientation angle of the target. Finally, an inference model in ONNX format is generated, and the TensorRT framework is used to accelerate model inference, outputting the 3D target detection sequence of 4 cameras.

[0079] Based on the Hungarian matching algorithm and combined with the internal and external parameter relationships of multiple roadside cameras, the sequence of 3D target detection frames is matched based on position, category, IOU, and re-matched.

[0080] S301: Perform 3D target detection on the collected multiple image data and output a 3D target detection frame, including using an improved target detection network ImVoxelNet to perform multi-category 3D target detection on the image to obtain a 3D target detection frame.

[0081] S302: Using the improved target detection network ImVoxelNet to perform multi-category 3D target detection on an image can be performed using the following steps:

[0082] Based on the original ImVoxelNet target detection network model, the model input image size is adjusted to achieve super-resolution detection and improve the accuracy of small target detection;

[0083] Modify ImVoxelNet to meet multi-category requirements, modify the anchor settings, set the anchor size based on the average size of the category, and better extract the features of each category;

[0084] Perform 3D convolution to enhance the model's 3D feature representation performance;

[0085] Finally, an inference model in ONNX format is generated, and the TensorRT framework is used to accelerate model inference and output 3D object detection sequences of multiple images.

[0086] S400: combining the internal and external reference relationship of multiple cameras at the intersection, matching based on position, category, matching based on IOU, and re-matching are performed on the sequence of 3D object detection frames;

[0087] S401: The following steps are adopted in combination with the internal and external parameter relationship of multiple cameras at the intersection:

[0088] Combined with the internal and external reference relationship of multiple roadside cameras, the sequences of multiple 3D target detection frames are matched based on position, category, IOU and re-matched, including the following steps:

[0089] Select two sequences from multiple 3D object detection box sequences, namely sequence a1 and sequence a2, and initialize the result sequence b;

[0090] The matching matrix M constructed from sequences a1 and a2 p And the final matching matrix M f, combined with the internal and external parameters of multi-camera calibration, the positions and orientation angles of the 3D object detection frames of the two sequences are converted from the camera coordinate system to the ENU coordinate system. If the center point of the 3D object detection frame of the a1 sequence falls within the 3D object detection frame of the a2 sequence, and the target classification results of the two are consistent, then the two 3D frames are preliminarily screened to meet the matching conditions; that is, if the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j If the matching condition is met, then M p (i, j) = 1, otherwise M p (i, j) = 0;

[0091] Convert the 3D object detection frames of the two sequences to the ENU coordinate system and calculate the intersection over union (IOU) between the 3D object detection frames of the two sequences, i.e., the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j The intersection-over-union ratio is expressed as IOU(i, j);

[0092] Traverse all 3D target detection boxes in the a1 sequence. If the i-th 3D box bbox3d i There is only one pre-matched 3D box, that is, only M p (i, j) = 1, then M f (i, j) = 1;

[0093] If M p (i, j) = 1 and M p If IOU(i, k)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(i, k), then M f (i, j) = 1, otherwise M f (i, k) = 1;

[0094] If M p (i, j) = 1 and the jth 3D box bbox3d j Already with the mth 3D box bbox3d of the a1 sequence m Match, that is, M p If Mf(m, j)=1 and Mf(i, j)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(m, j), then Mf(m, j)=1. f (i, j) = 1, the mth 3D box bbox3d of the a1 sequence m Repeat this step, otherwise M f (m, j) = 1, the i-th 3D box bbox3d of the a1 sequence i Repeat this step again;

[0095] Get the final matching matrix M f ;

[0096] According to the matching matrix M f , if M f m, n) = 1, then extract the mth 3D box bbox3d of the a1 sequence m and the nth 3D box bbox3d of the a2 sequence n The size information, position information, and orientation angle information of the target are weighted and added according to the detection confidence of the target, and the weighted target is incorporated into the result sequence b. If there is an unmatched target between the a1 sequence and the a2 sequence, the target is directly incorporated into the sequence b;

[0097] If the number of 3D object detection sequences is more than 2, then take a sequence a3 that has not been matched, and repeat the above "constructing the matching matrix M based on sequences a1 and a2" for sequence a3 and sequence b. p And the final matching matrix M f -According to the matching matrix M f , if M f (m, n) = 1" step, until there is no unmatched target sequence, and the final 3D target detection result sequence b is obtained.

[0098] S500: Output the entire intersection perception result based on the matching result to achieve collaborative output between multiple cameras. The result includes target category, geometric size, location information and orientation angle.

[0099] S501: The final merged 3D object detection result sequence is used as the unified perception result of the entire intersection, that is, the output of the collaborative work between multiple cameras at the intersection.

[0100] Furthermore, this embodiment also provides a 3D target detection implementation system based on intersection multi-camera collaboration, including:

[0101] The acquisition module uses GNSS timing to synchronize the time of multiple cameras at the intersection and collect image data from multiple cameras;

[0102] Construct the module, establish a unified spatial coordinate system, calibrate the internal and external parameters of the camera, convert each camera coordinate system into a unified ENU coordinate system, and realize the spatial alignment of multiple cameras at the intersection;

[0103] The detection module performs 3D object detection on the collected intersection multi-camera images and outputs the 3D object detection frame;

[0104] The matching module combines the internal and external reference relationships of multiple cameras at the intersection to match the sequence of 3D object detection frames based on position, category, IOU, and IOU.

[0105] The output module outputs the entire intersection perception result based on the matching results, realizing the coordinated output between multiple cameras. The results include target category, geometric size, location information and orientation angle.

[0106] This embodiment also provides a computer device, which is suitable for the implementation method of 3D target detection based on the collaboration of multiple cameras at intersections, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the implementation method of 3D target detection based on the collaboration of multiple cameras at intersections as proposed in the above embodiment.

[0107] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0108] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for realizing 3D target detection based on multi-camera collaboration at an intersection as proposed in the above embodiment is implemented.

[0109] In summary, the time synchronization of multiple cameras at the intersection is achieved through GNSS timing, which realizes the accurate synchronization of multi-camera data collection. This method solves the problem of inaccurate time synchronization in traditional multi-camera systems, ensures the accuracy of subsequent 3D target detection and fusion, and ultimately achieves the effect of improving the overall detection accuracy.

[0110] By establishing a unified ENU coordinate system and calibrating the camera's internal and external parameters, multi-camera spatial alignment is achieved. This step overcomes the difficulty of inconsistent coordinate systems for each camera in a traditional multi-camera system, laying the foundation for subsequent 3D target detection and multi-camera data fusion, ultimately achieving the effect of improving target positioning accuracy.

[0111] By improving the ImVoxelNet target detection network, multi-category 3D target detection is achieved for the captured images. This improvement includes adjusting the input image size, modifying the anchor settings, and performing 3D convolution, etc., which solves the problem that the traditional 2D detection method cannot obtain the target's 3D information, and ultimately achieves the effect of simultaneously obtaining the target category, geometric size, location information, and orientation angle, which is crucial for path planning and obstacle avoidance in intelligent transportation systems.

[0112] By combining the internal and external reference relationships of multiple cameras, the 3D target detection frame sequence is matched multiple times based on position, category and IOU, realizing the intelligent fusion of multi-camera data. This method solves the problems of occlusion and truncation of targets in single-camera detection, significantly improves the accuracy of target detection in complex scenes, and ultimately achieves the effect of expanding the perception range.

[0113] By outputting the unified perception results of the entire intersection, the coordinated output between multiple cameras is achieved. This method integrates information from multiple perspectives, overcomes the problem of the limited field of view of a single camera, and ultimately achieves the effect of fully perceiving the traffic conditions at the intersection, providing a more comprehensive and accurate decision-making basis for the intelligent transportation system.

[0114] In summary, the present invention not only solves the problem that traditional 2D target detection cannot accurately locate the target through the 3D target detection method of multi-camera collaboration, but also overcomes the limitations of single camera detection. This method not only improves the detection accuracy, but also significantly expands the perception range, providing more reliable data support for intelligent transportation systems. In particular, the method of the present invention can simultaneously obtain the 3D geometric information and orientation angle of the target, which is of great significance for the path planning and obstacle avoidance of intelligent networked vehicles, and is not achievable by traditional methods. In addition, the multi-camera collaborative mechanism of the method can effectively deal with the occlusion and truncation problems in complex traffic scenes, greatly improving the reliability and practicality of the system in practical applications. These effects are difficult to predict based on the existing technology, reflecting the innovativeness and practical value of the present invention.

[0115] Example 2

[0116] Reference Figure 2 - Figure 4 , which is the second embodiment of the present invention, provides a method for implementing 3D target detection based on multi-camera collaboration at an intersection. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0117] To verify the effectiveness and advantages of the present invention, a 30-day field test was conducted at a busy intersection in a large city. The intersection has an average daily traffic volume of about 50,000 vehicles and a pedestrian flow of about 20,000 people, making it an important node in urban traffic. The test system consists of 4 sets of intelligent roadside poles, each of which is equipped with a high-definition camera (resolution of 3840x2160), an edge computing unit (equipped with NVIDIA Jetson AGX Xavier module), a roadside communication unit and a GNSS receiver (u-blox ZED-F9P, with centimeter-level accuracy).

[0118] First, the GNSS receiver was used to synchronize the time of the four cameras. Through 30 consecutive days of data collection, the time synchronization error was controlled within 1ms, which is much better than the 10-50ms error range of the traditional NTP synchronization method. Then, the improved checkerboard calibration method was used to calibrate the internal parameters of the camera. In 20 sets of calibration images at different angles, the average reprojection error was only 0.15 pixels, which is a significant improvement over the 0.5 pixel error of the traditional method. The external parameter calibration used a test vehicle equipped with high-precision RTK to perform multiple round-trip movements at the intersection, and finally obtained the transformation matrix of the four cameras relative to the unified ENU coordinate system, and the average spatial alignment error was controlled within 5cm.

[0119] In the target detection stage, the improved ImVoxelNet network is used for 3D target detection. The network input size is adjusted to 1024x512, and the detection targets are divided into four categories according to the characteristics of the traffic scene: small cars, large cars, pedestrians and non-motor vehicles. By modifying the anchor settings and introducing 3D convolution and other optimization measures, the model's mAP on the KITTI dataset increased by 8.5 percentage points to 89.7%. After using TensorRT to accelerate the model, the processing time of a single image was reduced from the original 180ms to 65ms, meeting the real-time requirements.

[0120] In the multi-camera collaboration stage, the improved Hungarian algorithm is used for target matching. The occlusion and truncation problems are effectively solved through the position, category and IOU triple matching strategy. After 30 days of continuous testing, the system's average detection accuracy, recall rate and processing speed and other key indicators are shown in the following table:

[0121] Performance Indicators Traditional single camera 2D detection Method of the present invention Improvement Average detection precision (mAP) 76.2％ 92.8％ 21.8％ Average recall 72.5％ 94.3％ 30.1％ Position error (average) 0.75m 0.18m 76.0％ Heading angle error (average) 15.3° 3.2° 79.1％ Occluded target detection rate 45.6％ 87.9％ 92.8％ Processing speed (frames / second) 12 25 108.3％

[0122] By comparing the experimental data, the method of the present invention shows significant advantages and innovations in multiple key indicators. First, in terms of detection accuracy, the average detection accuracy (mAP) of this method reaches 92.8%, which is 21.8% higher than the traditional single-camera 2D detection method. This substantial improvement is mainly due to the combination of multi-camera collaboration and 3D target detection, which effectively overcomes the limitations of a single perspective. Especially in complex traffic scenes, multi-angle information fusion greatly reduces false detections and missed detections.

[0123] The improvement in average recall is even more significant, reaching 94.3%, 30.1% higher than the traditional method. This means that the system can detect more traffic participants, especially in crowded or partially blocked situations. This is of great significance for improving traffic safety and optimizing traffic flow management.

[0124] In terms of target positioning accuracy, the average position error of this method is only 0.18m, which is 76.0% higher than the 0.75m of the traditional method. This high-precision positioning capability is crucial for autonomous driving and intelligent traffic management systems, and can greatly improve the accuracy of vehicle obstacle avoidance and path planning. At the same time, the significant reduction in the heading angle error (from 15.3° to 3.2°, an increase of 79.1%) provides a more reliable basis for predicting the vehicle's motion trajectory.

[0125] Of particular note is the detection rate of occluded targets. In complex urban traffic environments, mutual occlusion between vehicles and pedestrians is a common problem. This method improves the detection rate of occluded targets from 45.6% to 87.9%, an increase of 92.8%. This breakthrough greatly enhances the reliability and safety of the system in practical applications.

[0126] In terms of processing speed, despite the use of more complex 3D detection algorithms and multi-camera data fusion, this method still increases the processing speed from 12 frames per second to 25 frames per second, achieving a 108.3% improvement. This is due to the optimized network structure and TensorRT acceleration, which enables the system to meet the needs of real-time processing and provide timely decision support for intelligent traffic control.

[0127] In summary, the method of the present invention has shown significant advantages in many aspects such as detection accuracy, positioning accuracy, occlusion processing capability and real-time performance. These improvements are not only the improvement of quantitative indicators, but more importantly, a qualitative leap. For example, high-precision 3D positioning and orientation detection provide intelligent transportation systems with unprecedented refined management capabilities. The effective detection of occluded targets greatly improves the adaptability and reliability of the system in complex urban environments. These innovative achievements have laid a solid technical foundation for the establishment of a safer and more efficient intelligent transportation system, and are expected to play a key role in the construction of future smart cities.

[0128] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A 3D target detection implementation method based on multi-camera collaboration at an intersection, characterized by: include, Multiple cameras at the intersection are synchronized through GNSS timing to collect image data from multiple cameras; Establish a unified spatial coordinate system, calibrate the internal and external parameters of the camera, convert each camera's coordinate system into a unified ENU coordinate system, and realize the spatial alignment of multiple cameras at the intersection; Perform 3D object detection on the collected intersection multi-camera images and output 3D object detection frames; Combined with the internal and external reference relationship of multiple cameras at the intersection, the sequence of 3D object detection frames is matched based on position, category, IOU and re-matched; The entire intersection perception result is output based on the matching results to achieve collaborative output between multiple cameras. The results include target category, geometric size, location information and orientation angle.

2. The method for realizing 3D target detection based on intersection multi-camera collaboration as claimed in claim 1, characterized in that: The time synchronization of multiple cameras at the intersection through GNSS timing includes synchronizing the cameras using GNSS satellite timing to ensure that the timestamps of data collected by different cameras at the same time are synchronized; Establish a unified ENU coordinate system, represent the same intersection area, select the ENU coordinate system as the reference coordinate system of multiple cameras, after calibration, the camera data can be directly converted to the ENU coordinate system, thereby achieving multi-camera space unification; Perform 3D target detection on the collected multiple image data and output the 3D target detection frame, including using the improved target detection network ImVoxelNet to perform multi-category 3D target detection on the image to obtain the 3D target detection frame.

3. The method for realizing 3D target detection based on multi-camera coordination at intersections as claimed in claim 2, characterized in that: The multi-camera at the intersection is a roadside camera, and the 3D target detection method represents the roadside collaborative perception of multiple cameras, multiple points, and multiple perspectives; The roadside camera is a high-definition camera; The multi-point position refers to multiple cameras installed and deployed at different locations of the intersection; Deploy multiple sets of intelligent roadside poles in the same intersection area, wherein the intelligent roadside poles have cameras, edge computing units, and roadside communication units; The multi-view refers to that multiple cameras deployed at different points or the same point have different perception ranges and perception perspectives, so as to expand the range of roadside perception.

4. The method for realizing 3D target detection based on intersection multi-camera collaboration as claimed in claim 3, characterized in that: The ensuring of synchronization of the timestamps of data collected by different cameras at the same time includes real-time transmission of the original video images of multiple cameras at the same intersection to the edge computing unit; The calibration represents that after the roadside camera is installed, the internal parameters of the cameras on different roadside poles are calibrated using the checkerboard calibration method; External parameter calibration is performed on different cameras respectively to obtain external parameter calibration matrices from multiple sets of camera coordinate systems to the ENU coordinate system, thereby achieving spatial alignment of cameras on different roadside poles.

5. The method for realizing 3D target detection based on intersection multi-camera collaboration as claimed in claim 4, characterized in that: The multi-category 3D target detection of images using the improved target detection network ImVoxelNet can be performed using the following steps: Based on the original ImVoxelNet target detection network model, the model input image size is adjusted to achieve super-resolution detection and improve the accuracy of small target detection; Modify ImVoxelNet to meet multi-category requirements, modify the anchor settings, set the anchor size based on the average size of the category, and better extract the features of each category; Perform 3D convolution to enhance the model's 3D feature representation performance; Finally, an inference model in ONNX format is generated, and the TensorRT framework is used to accelerate model inference and output 3D object detection sequences of multiple images.

6. The method for realizing 3D target detection based on multi-camera coordination at intersections as claimed in claim 5, characterized in that: The internal and external parameter relationship of combining multiple cameras at the intersection is achieved by the following steps: Combined with the internal and external reference relationship of multiple roadside cameras, the sequences of multiple 3D target detection frames are matched based on position, category, IOU and re-matched, including the following steps: Select two sequences from multiple 3D object detection box sequences, namely sequence a1 and sequence a2, and initialize the result sequence b; The matching matrix M constructed from sequences a1 and a2 p And the final matching matrix M f , combined with the internal and external parameters of multi-camera calibration, the positions and orientation angles of the 3D object detection frames of the two sequences are converted from the camera coordinate system to the ENU coordinate system. If the center point of the 3D object detection frame of the a1 sequence falls within the 3D object detection frame of the a2 sequence, and the target classification results of the two are consistent, then the two 3D frames are preliminarily screened to meet the matching conditions; that is, if the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j If the matching condition is met, then M p (i, j) = 1, otherwise M p (i, j) = 0; Convert the 3D object detection frames of the two sequences to the ENU coordinate system and calculate the intersection over union (IOU) between the 3D object detection frames of the two sequences, i.e., the i-th 3D frame bbox3d of the a1 sequence i The j-th 3D box bbox3d of the a2 sequence j The intersection-over-union ratio is expressed as IOU(i, j); Traverse all 3D target detection boxes in the a1 sequence. If the i-th 3D box bbox3d i There is only one pre-matched 3D box, that is, only M p (i, j) = 1, then M f (i, j) = 1; If M p (i, j) = 1 and M p If IOU(i, k)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(i, k), then M f (i, j) = 1, otherwise M f (i, k) = 1; If M p (i, j) = 1 and the jth 3D box bbox3d j Already with the mth 3D box bbox3d of the a1 sequence m Match, that is, M p If Mf(m, j)=1 and Mf(i, j)=1, then select the pair with the larger IOU for matching. That is, if IOU(i, j)>IOU(m, j), then Mf(m, j)=1. f (i, j) = 1, the mth 3D box bbox3d of the a1 sequence m Repeat this step, otherwise M f (m, j) = 1, the i-th 3D box bbox3d of the a1 sequence i Repeat this step again; Get the final matching matrix M f ; According to the matching matrix M f , if M f m, n) = 1, then extract the mth 3D box bbox3d of the a1 sequence m and the nth 3D box bbox3d of the a2 sequence n The size information, position information, and orientation angle information of the target are weighted and added according to the detection confidence of the target, and the weighted target is incorporated into the result sequence b. If there is an unmatched target between the a1 sequence and the a2 sequence, the target is directly incorporated into the sequence b; If the number of 3D object detection sequences is more than 2, then take a sequence a3 that has not been matched, and repeat the above "construct the matching matrix M between sequences a1 and a2" for sequence a3 and sequence b. p And the final matching matrix M f -According to the matching matrix M f , if M f (m, n) = 1" step, until there is no unmatched target sequence, and the final 3D target detection result sequence b is obtained.

7. The method for realizing 3D target detection based on multi-camera coordination at intersections as claimed in claim 6, characterized in that: The entire intersection perception result is output according to the matching result to achieve collaborative output between multiple cameras, and the result includes target category, geometric size, location information and orientation angle; The final merged 3D object detection result sequence is used as the unified perception result of the entire intersection, that is, the output of the collaborative work between multiple cameras at the intersection.

8. A 3D target detection implementation system based on the cooperation of multiple cameras at an intersection, based on the 3D target detection implementation method based on the cooperation of multiple cameras at an intersection according to any one of claims 1 to 7, characterized in that: Also includes, The acquisition module uses GNSS timing to synchronize the time of multiple cameras at the intersection and collect image data from multiple cameras; Construct the module, establish a unified spatial coordinate system, calibrate the internal and external parameters of the camera, convert each camera coordinate system into a unified ENU coordinate system, and realize the spatial alignment of multiple cameras at the intersection; The detection module performs 3D object detection on the collected intersection multi-camera images and outputs the 3D object detection frame; The matching module combines the internal and external reference relationships of multiple cameras at the intersection to match the sequence of 3D object detection frames based on position, category, IOU, and IOU. The output module outputs the entire intersection perception result based on the matching results, realizing the coordinated output between multiple cameras. The results include target category, geometric size, location information and orientation angle.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for realizing 3D target detection based on intersection multi-camera collaboration as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for realizing 3D target detection based on intersection multi-camera collaboration as described in any one of claims 1 to 7 are implemented.

Images