Target detection method, equipment, medium and terminal for multi-sensor information fusion

Abstract

The present invention belongs to the field of target detection technology and discloses a target detection method, device, medium and terminal using multi-sensor information fusion. By classifying a vision-based target detection algorithm, a CenterNet-based target detection algorithm is adopted, a frustum-based data association algorithm is selected to achieve coarse filtering of radar point clouds, and a Kalman filter algorithm is used to improve the accuracy of data association, and to perform spatiotemporal binding of different sensor data. The present invention improves the CenterNet target detector and the data association algorithm respectively. To address the problem of target detection loss in occluded scenes, the current position of the target is predicted by the offset of the target in consecutive frames. To address the problem of missed detection of small targets, a multi-scale information fusion method is proposed to improve the target detector. To address the problem of data association errors in dense scenes, a confidence data association method is proposed based on the improved target detector.

Description

Technical Field

[0001] The present invention belongs to the field of target detection technology, and in particular relates to a target detection method, device, medium and terminal for multi-sensor information fusion. Background Art

[0002] At present, target detection is widely used in video surveillance, intrusion detection, face recognition, autonomous driving and other fields. The use of deep learning to detect targets on camera images has made rapid progress, and the continuous improvement of detection rate and accuracy has promoted the application of target detection technology in various fields. However, under harsh weather conditions, the sensor quality of the camera is limited due to the increase of sensor noise in areas with sparse light and at night. Traditional target detection uses camera sensors to collect scene information, obtain the type of target of interest and locate the target position by processing the image information. In recent years, millimeter wave radar has been widely used in various target detection scenarios. Millimeter wave radar can sensitively detect the motion state of targets in the scene, and can obtain the speed and position information of moving targets without relying on any time information [5]. It is more robust and can work stably even in harsh environments such as fog and rain. Compared with camera sensors, the limitation of millimeter wave radar is that it cannot image and obtain the depth information of the target. Due to the different working principles and working ranges of camera and radar sensors, the use of a single sensor to achieve target detection has limitations. To address the above problems, the target detection algorithm that integrates radar and camera information can integrate the detection advantages of each sensor, improve the accuracy and robustness of target detection, and realize the prediction of target motion state.

[0003] CenterFusion (Center-based Radar and Camera Fusion for 3D Object Detection) is currently the industry's leading target detection algorithm that fuses radar and camera information. The algorithm consists of two stages: image-based target detection and information fusion. The first stage uses the CenterNet (Objects as Points) target detector to obtain target type and location information within the image. The second stage uses a frustum method to perform data association between radar and camera information. Finally, radar features are used to supplement image features to achieve 3D target detection and target motion state prediction. However, this algorithm still has the following issues: In dynamic scenes, targets are too small in the image and the mutual motion between objects causes them to occlude each other, resulting in target loss during image-based target detection. In scenes with dense targets, radar points of adjacent targets appear within the same frustum, resulting in data association errors when using frustum-based data association methods.

[0004] Through the above analysis, the problems and defects of the existing technology are as follows:

[0005] (1) In dynamic scenes, existing target detection methods suffer from target loss during the detection process.

[0006] (2) In target-dense scenarios, the existing target detection methods based on frustum data association methods may have data association errors.

[0007] (3) Object detection on camera images using deep learning has made rapid progress, and the continuously improving detection rate and accuracy have promoted the application of object detection technology in various fields. However, in harsh weather conditions, due to the increase in sensor noise in areas with sparse light and at night, the sensor quality of cameras is limited. Summary of the Invention

[0008] In response to the problems existing in the prior art, the present invention provides a target detection method, device, medium and terminal with multi-sensor information fusion, which can realize a target detection algorithm that fuses radar and camera information for three-dimensional target detection in dynamic scenes.

[0009] The present invention is implemented as follows: a target detection method based on multi-sensor information fusion includes: using a target detection algorithm based on vision through classification, adopting a target detection algorithm based on CenterNet, selecting a data association algorithm based on a frustum to achieve coarse filtering of radar point clouds, using a Kalman filter algorithm to improve the accuracy of data association, and performing spatiotemporal binding of different sensor data.

[0010] Furthermore, the specific steps of the target detection method of multi-sensor information fusion include:

[0011] Step 1: Vision-based object detection: Using an object detector with a spatiotemporal correlation mechanism, the current position of the target is predicted based on its spatial offset in consecutive frames. This step can effectively detect occluded targets.

[0012] Step 2: Radar and camera data association: Using a data fusion mechanism that correlates radar information from previous and subsequent frames, an improved visual target detector is used to obtain radar information for the same target in the previous frame. The radar value range for the target in the current frame is predicted using the radar information from the previous frame. Finally, a nearest neighbor association algorithm is used to obtain the most relevant radar points for data association. This step achieves temporal fusion and matching of heterogeneous radar and camera sensor data.

[0013] Step 3: This step uses the expanded frustum method to compensate for the inaccurate point cloud projection caused by the depth error obtained by the camera sensor and the radar detection;

[0014] Step 4: Sensor Data Association: When binding key image frames with radar point cloud information, coordinate conversion is first performed to unify the data collected by the millimeter-wave radar and camera in a standard coordinate system to achieve data association. This step achieves spatial fusion and matching of heterogeneous data from radar and camera sensors.

[0015] Step 5: Test Data Annotation: Use the radar sensor to collect target motion data. Through manual annotation, the image data is linked to the key radar data. The collected radar data includes raw data and data converted to a standard coordinate system. This step generates a test set and facilitates subsequent training and testing.

[0016] Step 6: Use the CenterFusion algorithm and the improved CenterFusion++ algorithm to achieve target detection effects on the current image, verify the proposed method of replacing the target detector, and use a single-target walking method to verify that replacing the CenterTrack++ target detector can reduce the ID switching frequency of the same target and keep the same target ID unchanged in consecutive frames. This step verifies the target detection function.

[0017] Furthermore, in step 1, in visual target detection, the CenteTrack++ target detector is selected to implement preliminary target detection, the FPN structure is used to improve the backbone network of the CenterTrack++ algorithm, and the feature pyramid is used to achieve information fusion of feature layers of different depths to obtain multi-scale features.

[0018] Furthermore, in the step 2 of associating the radar and camera information data, a Kalman filter algorithm is used to correct the actual measurement value of the sensor.

[0019] Furthermore, the sensor data association in step 4 involves conversions between five coordinate systems, namely, a standard coordinate system, a radar coordinate system, a camera coordinate system, an image coordinate system, and a pixel coordinate system;

[0020] The coordinate transformation includes:

[0021] (1) Transformation between standard coordinate system and camera coordinate system;

[0022] (2) Conversion between camera coordinate system and image coordinate system;

[0023] (3) Conversion between image coordinates and pixel coordinates.

[0024] Furthermore, the standard coordinate system is used to describe the position of the target in the real world and is represented by OW-XwYwZw, and the camera coordinate system is used to describe the positional relationship between the target and the camera sensor and is represented by OC-XCYCZC;

[0025] In the transformation between the standard coordinate system and the camera coordinate system, the rotation and translation relationships are combined to obtain the transformation between the camera coordinate system and the standard coordinate system:

[0026]

[0027] Where T represents the translation matrix used, R represents the rotation matrix, O represents the three-dimensional zero vector, and θ represents the rotation angle.

[0028] Furthermore, in the conversion between image coordinates and pixel coordinates, pixel coordinates are converted into image coordinates by translation, and the conversion process is expressed using a homogeneous coordinate system as follows:

[0029]

[0030] Where u represents the horizontal coordinate of the pixel coordinate system, v represents the vertical coordinate, (u0, v0) represents the coordinates of the origin in the pixel coordinate system, dx and dy represent the physical size of each pixel on the horizontal axis X and vertical axis Y respectively, and X and Y represent the horizontal coordinate and vertical coordinate of the image coordinate system.

[0031] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0032] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving these problems, we closely combine the technical solutions to be protected by the present invention and the results and data during the research and development process, and conduct a detailed and in-depth analysis of how the technical solutions of the present invention solve the technical problems and some creative technical effects brought about by solving the problems. The specific description is as follows:

[0033] This paper improves the CenterNet target detector and data association algorithm based on the CenterFusion algorithm framework. To address the problem of target detection loss in occlusion scenarios, it is proposed to use a target detector with a spatiotemporal association mechanism to predict the current position of the target through the offset of the target in consecutive frames. To address the problem of missed detection of small targets, it is proposed to improve the target detector using a multi-scale information fusion method. To address the problem of data association errors in dense scenarios, based on the improved target detector, a confidence data association method is proposed. That is, for the currently selected candidate box with higher confidence, the candidate box is weightedly fused based on its IOU with the surrounding candidate boxes and their respective category confidence information.

[0034] The vision-based target detection algorithm in the present invention is based on the CenterNet target detection algorithm. In response to the algorithm's target occlusion interference and small scale problems in dynamic scenes, combined with the idea of multi-layer feature fusion, a CenterTrack++ algorithm based on an improved feature extraction network is proposed. The algorithm first uses a target detector with a spatiotemporal correlation mechanism to predict the target's current position information based on the target's spatial offset in consecutive frames to achieve occluded target detection. By comparing the performance of different target detectors on the MOT17 dataset, the CenterTrack++ algorithm is selected to achieve image feature extraction. The Feature Pyramid Network (FPN) is then used to improve the CenterTrack++ algorithm. During the downsampling process of the backbone network, effective feature layers of different depths are extracted. After the convolution operation, they are fused with the FPN downsampling features to obtain a multi-scale feature fusion result, thereby improving the accuracy of small target detection.

[0035] The present invention effectively improves the accuracy of multi-source sensor data binding by selecting key points based on the Kalman filter algorithm. To avoid the phenomenon of real radar points being filtered out due to sensor measurement errors, the Kalman filter algorithm is introduced to realize radar information correction.

[0036] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0037] The present invention can realize the detection of occluded targets in dynamic scenes, improve the detection accuracy of small-scale targets, and solve the problem of data association errors caused by adjacent target radar points appearing in the same frustum in target-dense scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a flow chart of a target detection method using multi-sensor information fusion according to an embodiment of the present invention;

[0039] Figure 2 This is an overall framework diagram of the multi-sensor information fusion target detection method provided by an embodiment of the present invention;

[0040] Figure 3 is a flow chart of an improved target detection method with an association mechanism provided by an embodiment of the present invention;

[0041] Figure 4 This is a flow chart of radar correlation between front and back frames provided by an embodiment of the present invention;

[0042] Figure 5 Schematic diagram of a method for expanding a truncated cone provided by an embodiment of the present invention;

[0043] Figure 6 Schematic diagram of coordinate transformation relationship provided by an embodiment of the present invention;

[0044] Figure 7 Schematic diagram of the conversion between world coordinates and camera coordinates provided by an embodiment of the present invention;

[0045] Figure 8 Schematic diagram of the camera coordinate and image coordinate conversion model provided by an embodiment of the present invention;

[0046] Figure 9 Schematic diagram of image coordinate and pixel coordinate conversion provided by an embodiment of the present invention;

[0047] Figure 10 is a schematic diagram of a chessboard image set provided by an embodiment of the present invention;

[0048] Figure 11 This is a schematic diagram showing part of the data set provided by an embodiment of the present invention;

[0049] Figure 12 1 is a schematic diagram for comparing the effects provided by an embodiment of the present invention;

[0050] Figure 13 This is a diagram showing the effect of target detection in an occlusion scenario provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0052] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands upon the technical solutions of the claims.

[0053] like Figure 1 As shown, the target detection method of multi-sensor information fusion provided by the embodiment of the present invention includes:

[0054] S101: Vision-based object detection: Using an object detector with a spatiotemporal correlation mechanism, the target's current position information is predicted based on its spatial offset in consecutive frames to detect occluded objects.

[0055] S102: Radar and camera data association: This uses a data fusion mechanism that associates radar information from previous and subsequent frames. An improved visual target detector is used to obtain radar information for the same target in the previous frame. The radar value range of the target in the current frame is predicted using the radar information from the previous frame. Finally, a nearest neighbor association algorithm is used to obtain the most relevant radar points for data association.

[0056] S103: Use the extended frustum method to compensate for the inaccurate point cloud projection caused by the depth error obtained by the camera sensor and the radar detection;

[0057] S104: Sensor data association: In the process of binding the key image frame with the radar point cloud information, coordinate conversion is first performed to unify the data collected by the millimeter wave radar and the camera in a standard coordinate system to achieve data association;

[0058] S105: Test data annotation: Use the radar sensor to collect target motion data. Through manual annotation, the image data is bound to the key data of the radar data. The collected radar data includes the original data and the data converted to the standard coordinate system.

[0059] S106: Target detection function verification: Use the CenterFusion algorithm and the improved CenterFusion++ algorithm to achieve target detection effects on the current picture respectively, verify the proposed method of replacing the target detector, and use a single target walking method to verify that replacing the CenterTrack++ target detector can reduce the ID switching frequency of the same target and keep the same target ID unchanged in consecutive frames.

[0060] To address the problems of the CenterFusion target detection algorithm, the present invention proposes a target detection algorithm that fuses radar and camera information and can achieve three-dimensional target detection in dynamic scenes. Based on the CenterFusion algorithm framework, the CenterNet target detector and data association algorithm are improved respectively. To address the problem of target detection loss in occlusion scenes, the present invention proposes to use a target detector with a spatiotemporal association mechanism to predict the current position of the target through the offset of the target in consecutive frames. To address the problem of missed detection of small targets, the present invention proposes to improve the target detector using a multi-scale information fusion method. To address the problem of data association errors in dense scenes, the present invention proposes a confidence data association method based on the improved target detector.

[0061] 2. Application Examples: In order to demonstrate the creativity and technical value of the technical solution of the present invention, this section provides application examples of the claimed technical solution on specific products or related technologies.

[0062] Example 1

[0063] To achieve the detection of occluded targets: To solve the problem of target detection loss caused by target occlusion in dynamic scenes, the present invention proposes to use a target detector with a spatiotemporal correlation mechanism to predict the target's position information at the current moment based on the spatial offset of the target in consecutive frames.

[0064] Example 2

[0065] Completed heterogeneous data association of radar and camera information: This solved the problem of data association errors caused by adjacent target radar points appearing in the same frustum in dense target scenes, and solved the problem of real radar points being filtered out due to radar sensor measurement errors.

[0066] Example 3

[0067] The extended frustum is used to compensate for the inaccurate point cloud projection caused by the depth error obtained by the camera sensor and the radar detection.

[0068] Example 4

[0069] Sensor data association: This paper uses Zhang Zhengyou's camera annotation method to obtain the camera's internal and external parameters, and transforms the camera coordinate system into the image coordinate system and pixel coordinate system based on the camera's internal and external parameters. By establishing a world coordinate system, the radar and camera sensors achieve coordinate transformation;

[0070] Example 5

[0071] Coordinate transformation: Complete the transformation of the image coordinate system, camera coordinate system and pixel coordinate system in the three major coordinate systems.

[0072] Example 6

[0073] Camera sensor calibration: The key to converting the pixel coordinate system into the standard coordinate system is to obtain the internal and external parameters of the camera. This invention uses Zhang Zhengyou's camera annotation method.

[0074] Example 7

[0075] Test Data Annotation: Target motion data is collected using a radar sensor. Manual annotation is used to link the image data with key radar data. The collected radar data includes both raw data and data converted to a standard coordinate system. This step generates a test set and facilitates subsequent training and testing.

[0076] Example 8

[0077] Target detection function verification: The CenterFusion algorithm and the improved CenterFusion++ algorithm are used to achieve target detection effects on the current image, respectively. The proposed method of replacing the target detector is verified. A single-target walking method is used to verify that replacing the CenterTrack++ target detector can reduce the ID switching frequency of the same target and keep the same target ID unchanged in consecutive frames.

[0078] 3. Evidence of the effects of the embodiments: The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the existing technology. The following content describes them with reference to the data, charts, etc. of the experimental process.

[0079] The specific process of the target detection method of multi-sensor information fusion in the embodiment of the present invention is as follows:

[0080] Step 1: In terms of vision-based target detection: In order to solve the problem of target detection loss caused by target occlusion in dynamic scenes, the present invention proposes to use a target detector with a spatiotemporal correlation mechanism to predict the target's position information at the current moment based on the target's spatial offset in consecutive frames, thereby realizing the detection of occluded targets. By comparing the accuracy and ID switching rate of different target detectors in the MOT17 dataset, the CenteTrack++ target detector was finally selected to achieve preliminary target detection. In order to solve the problem of missed detection of targets due to the small size of the target in the image screen, it is proposed to use the FPN structure to improve the backbone network of the CenterTrack++ algorithm, and to achieve multi-scale features by fusing information of feature layers of different depths through the feature pyramid, thereby improving the detection accuracy of small-scale targets.

[0081] Step 2: Regarding radar and camera data association: To address the issue of data association errors caused by adjacent target radar points appearing within the same frustum in densely populated target scenarios, a data fusion mechanism is proposed to associate radar information from previous and subsequent frames. This mechanism uses an improved visual target detector to obtain radar information for the same target in the previous frame. The radar value range of the target in the current frame is predicted using the radar information from the previous frame. Finally, a nearest neighbor association algorithm is used to obtain the most relevant radar points for data association. To address the issue of true radar points being filtered out due to radar sensor measurement errors, a Kalman filter algorithm is proposed to correct the true sensor measurements.

[0082] Step 3: During the equipment installation process, slight spatial errors between the radar and camera sensors can cause positional discrepancies between the radar detection position and the target detection box obtained by the camera. This discrepancy becomes more pronounced when the radar point cloud data is projected onto the image. Frustum generation relies on the two-dimensional bounding box and depth information of the target obtained by the object detector. The target depth in the image features is not completely reliable, so the generated frustum may be smaller in scope, causing the actual radar point cloud data for the current target to appear outside the frustum. To address this issue, an extended frustum is used to compensate for the depth errors obtained by the camera sensor and the inaccurate point cloud projection caused by radar detection.

[0083] Step 4: Sensor data association. Due to the different principles of the millimeter wave radar and camera sensors, the data collected are independent of each other, that is, the two sensors use different coordinate systems in the process of obtaining information. Therefore, in the process of binding the key image frame with the radar point cloud information, it is necessary to first implement the coordinate conversion process. Only by unifying the data collected by the two sensors in the standard coordinate system can the data be associated. The data association process involves conversions between five coordinate systems, namely the standard coordinate system, radar coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system. The conversion relationship between these five coordinate systems is as follows: Figure 6 As shown, in the camera sensor, the present invention uses Zhang Zhengyou's camera annotation method to obtain the camera's internal and external parameters, and transforms the camera coordinate system into the image coordinate system and pixel coordinate system based on the camera's internal and external parameters. By establishing a world coordinate system, the radar and camera sensors achieve coordinate transformation.

[0084] Step 5: Coordinate transformation. In the process of coordinate transformation, the transformation between different coordinate systems is required depending on the different coordinate systems. (1) Transformation between the standard coordinate system and the camera coordinate system. The standard coordinate system is used to describe the position of the target in the real world. It is generally defined by the user and is represented by OW-XwYwZw. The camera coordinate system is used to describe the positional relationship between the target and the camera sensor. When the camera is fixed, the camera coordinate system is also fixed and is represented by OC-XCYCZC. Both the camera coordinate system and the standard coordinate system represent the three-dimensional coordinate system of the target in space. Therefore, there are only position and direction errors in the transformation process. The transformation between the two can be achieved by combining rotation and translation. The present invention takes rotation around the X-axis as an example. Figure 7 Show the conversion relationship between the two.

[0085] Figure 7 The black coordinate system represents the camera coordinate system, and the yellow coordinate system represents the standard coordinate system. When rotating a model around the X axis, only the coordinate conversion between the Y axis and the Z axis needs to be considered. The rotation angle is represented by θ. The red point represents the point P that needs to be converted. Let the coordinate of this point in the camera coordinate system be (Xw, Yw, Zw). The X axis coordinate will not change, and the Y axis can be obtained by adding auxiliary lines. w =cosθ·Y C +sinθ,Z w = -sinθ·Y C +cosθ·Z C ;

[0086] The vector form is expressed as:

[0087]

[0088] Similarly, the conversion formula for rotation around the Y axis is:

[0089]

[0090] The conversion formula for rotation around the Z axis is:

[0091]

[0092] In addition to considering the rotation factor, it is also necessary to consider the translation relationship between the two coordinate systems. The translation transformation is as follows:

[0093]

[0094] Combining the rotation and translation relationships can obtain the transformation between the camera coordinate system and the standard coordinate system:

[0095]

[0096] Where T represents the translation matrix used, R represents the rotation matrix, and O represents the three-dimensional zero vector.

[0097] (2) Conversion between camera coordinate system and image coordinate system. The conversion model between camera coordinate system and image coordinate system is as follows: Figure 8 shown.

[0098] (3) Conversion between image coordinates and pixel coordinates. The coordinates of the pixel coordinate system are all expressed in the form of pixels, but the image coordinate system is expressed in the form of physical distance, and the unit is usually millimeters. The pixel coordinate system is not conducive to coordinate conversion, so it is necessary to convert pixel coordinates into image coordinates. The mathematical model is as follows Figure 9 As shown in the figure, O-uv represents the pixel coordinate system, and O1-xy represents the image coordinate system. The image coordinate system is parallel to the pixel coordinate system, so in the conversion process, the pixel coordinates can be converted to the image coordinates only by translation. The conversion process is expressed in the homogeneous coordinate system as:

[0099]

[0100] Step 6: Camera sensor calibration. Converting the pixel coordinate system to a standard coordinate system is crucial for determining the camera's intrinsic and extrinsic parameters. This method uses Zhang Zhengyou's camera calibration method, capturing at least 10 images of the checkerboard from different angles and distances. The camera's intrinsic and extrinsic parameters are obtained using the OpenCV visual processing library. Figure 10Represents a partial set of images taken of a chessboard. During the process of photographing the chessboard, the chessboard is photographed from different angles and under different lighting perceptions. The images taken by the Hikvision camera used in the present invention can obtain different resolutions. To obtain better clarity, the main code stream is used to complete the image acquisition, and the resolution of the acquired images is 1280×960. Then, Zhang Dingyou's camera annotation mode is used to traverse the images in the current folder, and the camera's intrinsic parameters, distortion parameters, rotation parameters, and translation parameters are finally calculated, and finally the corrected camera intrinsic parameters are obtained.

[0101] Step 7: Test data annotation. During the algorithm testing process, in order to better test the effectiveness of this algorithm, the camera sensor is used to collect 12 seconds of continuous motion video of two targets. The video of Hikvision camera consists of 25 frames per second, and the current motion video is converted into 300 frames of pictures. During the target movement, the radar sensor is used to collect target motion data. The radar data can obtain 10 frames of data in 1 second during the collection process. The binding of image data and radar data key data is achieved through manual annotation. Part of the collected data set is shown as follows Figure 11 The collected video data is converted into each frame of picture display, and the numbering is completed starting from the first frame. Figure 11 (b) shows the moving target data collected within a specific second. This data includes the target ID, X-axis offset relative to the radar, Y-axis offset, Z-axis height (the radar sensor cannot collect height information; the default is 0), left-right speed, and forward-backward speed. The collected radar data includes both raw data and data converted to a standard coordinate system.

[0102] Step 8: Verify the target detection function. After establishing the standard coordinate system, creating the radar and camera sensors, solving the camera sensor's internal and external parameters, and binding the data, the CenterFusion algorithm and the improved CenterFusion++ algorithm are used to achieve target detection in the current frame. First, the proposed target detector replacement method is verified. During the comparative test, a single target walking method is used to verify that replacing the CenterTrack++ target detector effectively reduces the ID switching frequency of the same target and maintains the same target ID in consecutive frames.

[0103] like Figure 12(a) in the figure shows target recognition using the CenterNet object detector. This paper selects four consecutive frames where ID switching occurs. In frames 159 and 160, the current object detector still identifies the same target with ID 1. In frame 161, an ID switch occurs, and the object detector simultaneously displays two target detection frames, one with ID 1 and one with ID 2, on the screen. When the process reaches frame 162, an ID switch occurs, and the ID of the same target is set to 2. Figure 12 (b) shows the recognition effect when the target detector is replaced with the CenterTrack++ algorithm and the same four frames are selected. In this case, the ID does not switch, indicating that changing the target detector can reduce the ID switching of the same target.

[0104] Figure 13 This method fuses radar sensor data with image information to detect occluded targets in consecutive frames, while preserving the ID of the same target. To verify that the proposed algorithm can maintain the ID of the same target, we use intermittent frames to demonstrate the effect. Figure 13 (a) in the figure indicates that at the 103rd frame, three unobstructed targets can be detected. Figure 13 (b) and (c) show the detection results of frames 105 and 107. At this time, the same ID of the target can be maintained without switching, and occlusion between targets begins to appear in frame 107. Figure 13 (d) shows an object occlusion scenario in frame 109, where the black shirt object (numbered 4) is occluded by the first two objects. However, the object detection algorithm of our invention is still able to detect the occluded object and maintain its ID unchanged. This demonstrates the effectiveness of the CenterFusion++ algorithm proposed in this paper in real-world scenarios.

[0105] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0106] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A target detection method based on multi-sensor information fusion, characterized in that: The target detection method of multi-sensor information fusion includes: By classifying the vision-based target detection algorithm, adopting the CenterNet-based target detection algorithm, selecting the frustum-based data association algorithm to achieve coarse filtering of the radar point cloud, using the Kalman filter algorithm to improve the accuracy of data association, and performing spatiotemporal binding of different sensor data; The target detection method of multi-sensor information fusion includes the following specific steps: Step 1: Vision-based object detection: Using an object detector with a spatiotemporal correlation mechanism, the target's current position is predicted based on its spatial offset in consecutive frames, enabling detection of occluded objects. Step 2: Radar and camera data association: This involves a data fusion mechanism that associates radar information from previous and subsequent frames. An improved visual target detector is used to obtain radar information for the same target in the previous frame. The radar value range for the target in the current frame is predicted using the radar information from the previous frame. Finally, a nearest neighbor association algorithm is used to obtain the most relevant radar points for data association. Step 3: Use the extended frustum method to compensate for the inaccurate point cloud projection caused by the depth error obtained by the camera sensor and the radar detection; Step 4: Sensor Data Association: When binding key image frames with radar point cloud information, coordinate conversion is first performed to unify the data collected by the millimeter-wave radar and camera in a standard coordinate system to achieve data association. Step 5: Test Data Annotation: Use the radar sensor to collect target motion data. Through manual annotation, the image data is bound to the key data of the radar data. The collected radar data includes the original data and the data converted to the standard coordinate system. Step 6: Verify the target detection function: Use the CenterFusion algorithm and the improved CenterFusion++ algorithm to achieve target detection effects on the current image, verify the proposed method of replacing the target detector, and use a single target walking method to verify that replacing the CenterTrack++ target detector can reduce the ID switching frequency of the same target and keep the same target ID unchanged in consecutive frames.

2. The target detection method of multi-sensor information fusion according to claim 1, characterized in that: In step 1, in visual target detection, the CenteTrack++ target detector is selected to perform preliminary target detection, the FPN structure is used to improve the backbone network of the CenterTrack++ algorithm, and the feature pyramid is used to fuse information of feature layers of different depths to obtain multi-scale features.

3. The target detection method of multi-sensor information fusion according to claim 1, characterized in that: In the step 2 of associating the radar and camera information data, the Kalman filter algorithm is used to correct the actual measurement value of the sensor.

4. The target detection method of multi-sensor information fusion according to claim 1, characterized in that: The sensor data association in step 4 involves conversion between five coordinate systems, namely the standard coordinate system, the radar coordinate system, the camera coordinate system, the image coordinate system and the pixel coordinate system; The coordinate transformation includes: (1) Transformation between standard coordinate system and camera coordinate system; (2) Conversion between camera coordinate system and image coordinate system; (3) Conversion between image coordinates and pixel coordinates.

5. The target detection method of multi-sensor information fusion according to claim 4, characterized in that: The standard coordinate system is used to describe the position of the target in the real world and is represented by OW-XwYwZw. The camera coordinate system is used to describe the positional relationship between the target and the camera sensor and is represented by OC-XCYCZC. In the transformation between the standard coordinate system and the camera coordinate system, the rotation and translation relationships are combined to obtain the transformation between the camera coordinate system and the standard coordinate system: ， Where T represents the translation matrix used, R represents the rotation matrix, O represents the three-dimensional zero vector, and θ represents the rotation angle.

6. The target detection method of multi-sensor information fusion according to claim 4, characterized in that: In the conversion between image coordinates and pixel coordinates, pixel coordinates are converted into image coordinates by translation. The conversion process is expressed using a homogeneous coordinate system as follows: 。 7. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the target detection method of multi-sensor information fusion according to any one of claims 1 to 6.

8. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the target detection method according to any one of claims 1 to 6 based on multi-sensor information fusion.

9. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the steps of the target detection method of multi-sensor information fusion as described in any one of claims 1 to 6.

Images