A multi-target detection method

Abstract

The present invention relates to a multi-target detection method, comprising: acquiring image data and radar data, and performing data calibration; obtaining visual feature vectors and radar feature vectors based on the calibrated image data and radar data; performing fuzzy inference on the visual feature vectors and radar feature vectors using a pre-trained fuzzy neural network model, and outputting fuzzy inference results, the fuzzy inference results including the degree of match between each target in the image data and the radar data; and screening out targets whose matching degree exceeds a preset threshold from the fuzzy inference results. Compared with the prior art, the present invention can improve detection accuracy and efficiency while reducing hardware costs. It is suitable for cycling traffic and solves the problems of low detection accuracy, high hardware costs, insufficient real-time performance, and lack of non-motor vehicle-specific target classification in complex environments caused by traditional fusion methods. It improves cyclists' safety perception capabilities in complex traffic scenarios and has the advantages of real-time and interpretability.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a multi-target detection method. Background Art

[0002] In recent years, non-motorized modes of transportation, represented by bicycles and electric bicycles, have rapidly developed globally, and the number of cyclists continues to expand. However, the resulting traffic safety risks cannot be ignored. Cycling safety issues are increasingly prominent due to perception delays or blind spots, especially in complex traffic scenarios such as at night or in inclement weather. Accident casualties are high, and there is an urgent need to improve perception capabilities.

[0003] In the field of target perception and detection in cycling traffic scenarios, multi-sensor data fusion strategies effectively improve detection accuracy by integrating complementary information. Current mainstream solutions generally utilize a heterogeneous sensor combination of radar and cameras. Radar offers all-weather capability and precise distance and speed measurement, but also faces challenges such as sparse point clouds and specular reflections. Cameras, on the other hand, can identify targets such as bicycles and electric vehicles through image texture features, but are prone to missed or false detections at night in low illumination or in inclement weather such as rain or snow, and cannot provide accurate three-dimensional spatial positioning. The synergistic fusion of these two approaches can create an all-weather perception system that complements radar's motion parameter detection characteristics with visual target recognition capabilities, thereby reducing the risk of traffic accidents.

[0004] However, current object detection methods that use radar and camera data fusion still have some drawbacks. For example, Zhou et al. proposed a method for achieving feature-level fusion of radar and camera data in a bird's-eye view (see Zhou, X., et al., "Radar and Camera Fusion in Bird's Eye View for Autonomous Driving," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 5, 2022, pp. 1234-1245). This method uses radar to obtain target range and velocity information while simultaneously extracting image features from the camera. These two types of data are then projected onto the bird's-eye view plane for fusion, and finally, a deep learning model is used to achieve object detection. However, this method relies on high-precision sensors and powerful computing platforms, resulting in high hardware costs and complex deployment, making it unsuitable for resource-constrained non-motorized vehicle scenarios. Furthermore, its algorithm is complex and interpretable, and it suffers from insufficient real-time performance, making it unable to meet the fast response requirements of cycling scenarios.

[0005] For example, Xu et al. proposed an attention-based adaptive fusion network (AFnet), which fuses radar and camera data through a deep learning model. This method, trained on the NuScenes and CARLA datasets, aims to improve object detection accuracy. (Reference: Xu C, Zhao H, Xie H, et al. Multi-sensor Decision-level Fusion Network Based on Attention Mechanism for Object Detection[J].IEEE Sensors Journal, 2024.) However, this method relies on large-scale datasets for model training, has a complex network structure, and requires a high-performance GPU for real-time inference. This significantly increases hardware costs, making it difficult to deploy in resource-constrained non-motorized vehicle scenarios. Furthermore, the high computational complexity of the model limits inference speed, making it unable to meet the real-time requirements of non-motorized vehicle multi-object detection systems. Furthermore, the system's reliance on high-precision radar and camera sensors further increases costs and deployment difficulties, making it difficult to meet low-cost constraints.

[0006] For example, Alai H. and Rajamani R. proposed a system for detecting and tracking target vehicles by fusing data from low-cost 2-D radar and a monocular camera. This system locates the target vehicle's corners by analyzing the radar's spatial data and the camera's image information, inferring the vehicle's posture. The system then uses a high-gain observer to process noise in the data, thereby stably calculating the target vehicle's position, velocity, and direction. However, the data points generated by the 2-D radar are relatively sparse, and in complex environments, they may not fully capture the target, resulting in incomplete detection results. Furthermore, while the high-gain observer algorithm improves accuracy, it is computationally intensive, potentially affecting the system's real-time responsiveness. Furthermore, in rainy, foggy, or low-light conditions, the monocular camera's performance degrades, further impacting the detection accuracy of the entire system.

[0007] In summary, the existing target detection solutions that integrate radar and camera data mainly have the problems of high hardware deployment cost, low detection accuracy and efficiency. Summary of the Invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a multi-target detection method that can improve detection accuracy and detection efficiency while reducing hardware costs.

[0009] The purpose of the present invention can be achieved by the following technical solution: A multi-target detection method comprises the following steps:

[0010] S1. Acquire image data and radar data and perform data calibration;

[0011] S2. Obtaining a visual feature vector and a radar feature vector based on the image data and radar data after data calibration;

[0012] S3. Performing fuzzy inference on the visual feature vector and the radar feature vector using a pre-trained fuzzy neural network model, and outputting a fuzzy inference result, wherein the fuzzy inference result includes a matching degree between the image data and each target in the radar data;

[0013] S4. Filter out targets whose matching degree exceeds a preset threshold from the fuzzy inference results.

[0014] Furthermore, in step S1, the image data is acquired by a camera, and the radar data is acquired by a radar. The data calibration in step S1 includes spatial alignment and temporal alignment of the data. The spatial alignment is specifically performed by correcting the yaw angle and pitch angle of the camera and the radar, thereby unifying the image data and the radar data into the same coordinate system, thereby achieving spatial alignment of the data.

[0015] Time alignment involves synchronizing the camera and radar through an external trigger signal, sorting the data by timestamp, setting a time window based on the camera's timestamp, and selecting the radar data with the smallest time difference to align with the image data.

[0016] Alternatively, you can use the radar timestamp as the reference, set the time window, and select the image data with the smallest time difference to align with the radar data.

[0017] Furthermore, the specific process of correcting the yaw angle and pitch angle of the camera and radar is as follows:

[0018] According to the radar yaw angle output by the radar and radar elevation angle , determine the camera yaw angle of the camera by the following formula And the camera pitch angle :

[0019]

[0020]

[0021]

[0022] in,( ， ) is the center pixel coordinate of the target frame in the camera, 、 is the focal length of the camera, 、 is the optical center of the camera, ( ， ) is the coordinate of the target in the normalized plane;

[0023] Then the difference between the radar yaw angle and the camera yaw angle is calculated As a compensation value for correcting the radar yaw angle, the difference between the radar pitch angle and the camera pitch angle is As a compensation value for correcting the radar pitch angle, the image data and radar data are unified into the camera coordinate system;

[0024] Or vice versa, the image data and radar data are unified into the radar coordinate system.

[0025] Furthermore, the step S2 specifically processes the image data using the YOLO algorithm to obtain a visual feature vector suitable for fuzzy neural network input. ;

[0026] And perform point cloud data preprocessing on radar data to obtain radar feature vector .

[0027] Furthermore, the visual feature vector x i Specifically:

[0028] ,

[0029] in, Indicates the image data goals, ( ， ) is the first The coordinates of the target, and are the width and height of the target bounding box, respectively. is the target category, is the confidence level of the target;

[0030] The radar characteristic vector y j Specifically:

[0031] ,

[0032] in, Indicates the radar data goals, ( ， ) is the first The coordinates of the target, is the target distance detected by the radar, is the radar yaw angle, is the target speed detected by the radar, It is the target movement trend detected by radar.

[0033] Furthermore, the pre-trained fuzzy neural network model in step S3 includes an input layer, a fuzzification layer, a rule layer, a normalization layer and an output layer;

[0034] The input layer is used to input the visual feature vector and the radar feature vector into the fuzzification layer;

[0035] The fuzzification layer is used to semantically describe the input quantity through the membership function to map the input quantity to the fuzzy set;

[0036] The rule layer is used to reason and calculate the fuzzified input according to the fuzzy rules;

[0037] The normalization layer is used to convert the results of fuzzy reasoning and calculation into accurate fuzzy scores;

[0038] The output layer is used to output the matching degree between the image data and each target in the radar data.

[0039] Furthermore, the specific working process of the fuzzification layer is as follows:

[0040] For the image data Target and radar data The distance between the targets Describe:

[0041] ,

[0042] We obtain three fuzzy sets: matching, partial matching, and non-matching, which are described using piecewise linear functions:

[0043] ;

[0044] in, is the fuzzy score corresponding to the distance fuzzy matching;

[0045] Speed ​​of target detected by radar Describe and obtain 5 fuzzy sets: low, slightly low, medium, slightly fast, fast, and use piecewise linear function to describe:

[0046] ,

[0047] ,

[0048] ,

[0049] ,

[0050] ;

[0051] Among them, 、 、 、 、 are piecewise linear functions corresponding to the 5 fuzzy sets of "low, slightly low, medium, slightly fast, fast" respectively;

[0052] The reason for dividing the target speed detected by the radar into five fuzzy sets is to effectively reflect the motion characteristics of different targets. According to traffic regulations, the maximum speed of an electric bicycle cannot exceed 25 km / h , and the speed of a motor vehicle on an urban road cannot exceed 50 km / h, Therefore, intervals are divided as 0~15 km / h , 10~25 km / h , 20~35 km / h , 30~45 km / h , 40~45+ km / h For this interval, when is less than 10 v , the speed is considered small, so the fuzzy membership degree is 1. When 10 < km / h ≤15, the membership degree of the speed for this fuzzy set is linearly described by the formula v . When >15, the membership degree is 0, which conforms to the triangular membership function in the fuzzy neural network; v Similarly, when <40, the membership degree is 0. When 40 < v ≤ 45, the membership degree of the speed for this fuzzy set is linearly described by the formula v . When <00,00190>>45, the membership degree is 1; for 、 、 、 these three intervals, the trapezoidal membership function in the fuzzy neural network is used to achieve smooth transition through linear increase / decrease for easy description;

[0053] By describing the change rate of the target bounding box in the image data, 2 fuzzy sets are obtained: approaching, departing, and are described using the following membership functions:

[0054] , ,

[0055] in, is the membership function corresponding to the “close” fuzzy set, is the membership function corresponding to the “away” fuzzy set;

[0056] Since the rate of change of the target bounding box It may be affected by factors such as lighting and occlusion, so the Gaussian membership function in the fuzzy neural network is used to smooth these uncertainties to effectively handle small differences in the bounding box change rate.

[0057] Furthermore, the fuzzy rules include:

[0058] Rule 1: If "Speed ​​= Fast" and "Radar Movement Trend = Approaching" and "Target Type = Car", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0059] ;

[0060] in, is the fuzzy score of rule 1, The corresponding value is "speed = fast". The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = car";

[0061] Rule 2: If "Speed ​​= Slightly Faster" and "Radar Movement Trend = Approaching" and "Target Type = Electric Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0062] ;

[0063] in, is the fuzzy score of rule 2, The corresponding value is "speed = slightly faster". The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = electric bicycle";

[0064] Rule 3: If "Speed ​​= Low" and "Radar Movement Trend = Far Away" and "Target Type = Electric Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0065] ;

[0066] in, is the fuzzy score of rule 3, The corresponding value is "speed = low". The corresponding value is "radar movement trend = far away". The corresponding value is "target type = electric bicycle";

[0067] Rule 4: If "Speed ​​= Slightly Low" and "Radar Movement Trend = Approaching" and "Target Type = Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0068] ;

[0069] in, is the fuzzy score of rule 4, The corresponding value is "speed = slightly lower". The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = bicycle";

[0070] Rule 5: If "Speed ​​= Medium" and "Radar Movement Trend = Far Away" and "Target Type = Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0071] ;

[0072] in, is the fuzzy score of rule 5, The corresponding value is "speed = medium". The corresponding value is "radar movement trend = far away". The corresponding value is "target type = bicycle";

[0073] Rule 6: If "Speed ​​= Fast" and "Radar Movement Trend = Far Away" and "Target Type = Pedestrian", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0074] ;

[0075] in, is the fuzzy score of rule 6, The corresponding value is "speed = fast". The corresponding value is "radar movement trend = far away". The corresponding value is "target type = pedestrian";

[0076] Rule 7: If "Radar Movement Trend = Approaching" and "Visual Detection Movement Trend = Approaching", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0077] ;

[0078] in, is the fuzzy score of rule 7, The corresponding value is "visual detection movement trend = approaching". The corresponding value is "radar movement trend = approaching";

[0079] Rule 8: If "Radar Movement Trend = Far Away" and "Visual Detection Movement Trend = Far Away", then "Match Probability = High". The corresponding reasoning and calculation formula is:

[0080] ;

[0081] in, is the fuzzy score of rule 8, The corresponding value is "Visual detection movement trend = away". The corresponding value is “radar movement trend = away”.

[0082] The above rules 1 to 6 are based on the target's fuzzy speed and radar motion trend. , and the target type judged by the visual module In the design, according to experience, the target that can approach at a high speed is a car accelerating from behind, so the design rule 1 is adopted; the target that can approach at a relatively high speed is an electric vehicle accelerating from behind, so the design rule 2 is adopted; the target that can move away at a low speed is an electric vehicle with a speed slightly lower than itself from behind, so the design rule 3 is adopted; the target that can approach at a relatively low speed is a bicycle accelerating from behind, so the design rule 4 is adopted; the target that can move away at a medium speed is a bicycle with a speed lower than itself from behind, so the design rule 5 is adopted; the target that moves away from the rear at a high speed is a pedestrian with a speed much lower than itself (the relative speed between the two is large), so the design rule 6 is adopted.

[0083] The above rules 7 and 8 are based on the movement trend of radar feedback. The fuzzy motion trend obtained based on the target frame change rate Design is performed so that when both are close, they are considered a match; when both are far away, they are considered a match.

[0084] Furthermore, the fuzzy score output by the normalization layer is specifically:

[0085] ,

[0086] ,

[0087] ,

[0088] in, It is i The visual target and j The fuzzy score obtained by matching between radar targets, 、 、 Distance fuzzy matching , speed fuzzy matching , motion trend fuzzy matching The activation weights of these three types of fuzzy matching results are: is to select the maximum value among the fuzzy scores obtained from rules 1 to 6, The maximum value is selected from the fuzzy scores obtained from rule 7 and rule 8.

[0089] Furthermore, the output layer is specifically based on the fuzzy score and the multi-target matching matrix, and outputs a matching matrix between the image data and each target in the radar data.

[0090] Compared with the prior art, the present invention has the following advantages:

[0091] After acquiring image and radar data, the present invention first performs data calibration. Based on the calibrated image and radar data, visual and radar feature vectors are obtained. Fuzzy inference is then performed on the visual and radar feature vectors using a pretrained fuzzy neural network model, outputting a fuzzy inference result. The fuzzy inference result includes the degree of match between each target in the image and radar data. Finally, targets with a match exceeding a preset threshold are screened from the fuzzy inference result. This method uses a fuzzy neural network to fuse the visual and radar feature vectors, fuzzifying the quantized input features and performing matching based on fuzzy rules. This method is highly interpretable and can effectively reduce the input data required for model training. Furthermore, the fuzzy neural network has fewer layers than deep learning frameworks such as convolutional neural networks, resulting in a simpler network structure and significantly lower computing power requirements than other algorithms. This makes it suitable for edge computing platforms with limited computing power, effectively reducing deployment costs. This invention effectively addresses the problems of traditional fusion methods, such as low detection accuracy in complex environments, high hardware costs, insufficient real-time performance, and a lack of specific non-motor vehicle classification.

[0092] The present invention performs data calibration on image data and radar data, including spatial alignment and temporal alignment. Spatial alignment corrects the yaw and pitch angles of the camera and radar, thereby unifying the image and radar data into the same coordinate system and achieving spatial alignment. Temporal alignment sorts the data by timestamp, sets a time window based on the camera's timestamp, and selects the radar data with the smallest time difference for time alignment with the image data. Conversely, sets a time window based on the radar's timestamp and selects the image data with the smallest time difference for time alignment with the radar data. This data calibration ensures that subsequent processing accurately obtains visual feature vectors and radar feature vectors.

[0093] On the one hand, the present invention processes image data through the YOLO algorithm to obtain visual feature vectors suitable for fuzzy neural network input, which can greatly improve detection efficiency while ensuring accuracy as much as possible, reducing the computing power requirements of the deployment platform; on the other hand, point cloud data preprocessing is performed on radar data to obtain radar feature vectors, which can extract key features, identify and eliminate outliers or noise data, and improve the detection accuracy of low-cost radar.

[0094] In the present invention, the pre-trained fuzzy neural network model includes an input layer, a fuzzification layer, a rule layer, a normalization layer, and an output layer. The fuzzification layer semantically describes the input quantity through a preset membership function and maps the input quantity to a fuzzy set. The rule layer infers and calculates the fuzzified input quantity according to fuzzy rules. The normalization layer converts the results of the fuzzy reasoning and calculation into an accurate fuzzy score. Finally, the output layer outputs the degree of match between the image data and each target in the radar data. The present invention utilizes fuzzy neural networks and weight distribution to fuse visual and radar data, introduces expert knowledge to design membership functions, compiles fuzzy rules, fuzzifies quantized input features, and matches them using fuzzy rules. This method can achieve highly robust and low-cost target detection and recognition, and has the advantages of real-time and interpretability. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 Schematic diagram of the method flow of the present invention;

[0096] Figure 2 Schematic diagram of the application process of the embodiment;

[0097] Figure 3 Schematic diagram of the field of view of the camera and radar in the embodiment;

[0098] Figure 4 Schematic diagram of the working process of the fuzzy neural network model in the embodiment;

[0099] Figure 5 Schematic diagram comparing the data fusion time of this solution and other algorithms in the embodiment. DETAILED DESCRIPTION

[0100] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0101] Example

[0102] like Figure 1 As shown, a multi-target detection method includes the following steps:

[0103] S1. Acquire image data and radar data and perform data calibration;

[0104] S2. Obtaining a visual feature vector and a radar feature vector based on the image data and radar data after data calibration;

[0105] S3. Performing fuzzy inference on the visual feature vector and the radar feature vector using a pre-trained fuzzy neural network model, and outputting a fuzzy inference result, wherein the fuzzy inference result includes a matching degree between the image data and each target in the radar data;

[0106] S4. Filter out targets whose matching degree exceeds a preset threshold from the fuzzy inference results.

[0107] This embodiment applies the above solution, such as Figure 2 As shown, the main contents are:

[0108] First, image data is acquired through a camera, and radar data is acquired through a radar. In this embodiment, a millimeter-wave radar is used. The radar is installed under the seat of a non-motorized vehicle or bicycle, perpendicular to the ground and ensuring that there is no obstruction. The radar can directly output the yaw angle, pitch angle, speed, distance and radar movement trend (approaching / far away) of the target it detects. The camera is also installed under the seat of the non-motorized vehicle or bicycle, and is installed in parallel with the radar. The camera's field of view covers 120° to the rear, such as Figure 3 As shown in the figure, the camera outputs the type of target it detects (vehicle, bicycle, electric bicycle, pedestrian, etc.), confidence (0~1) and target box area change rate through the YOLO (such as YOLOv8n) algorithm.

[0109] In this embodiment, the multi-target detection method is implemented by a software or hardware device, including but not limited to at least one of the following: a user device, a network device, etc. User devices may include but are not limited to computers, smartphones, personal digital assistants (PDAs), and other electronic devices. Network devices may include but are not limited to a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of computers or network servers. Cloud computing is a type of distributed computing, consisting of a group of loosely coupled computers forming a super virtual computer.

[0110] This embodiment specifically implements the multi-target detection method described above using an edge computing platform, such as the RK3576 embedded platform. This platform runs a fuzzy neural network (FNN) model and the YOLOv8n algorithm. This platform processes data captured by cameras and radar, and outputs matching results for targets in both data types. Targets whose matches exceed a preset threshold are identified as real targets; otherwise, they are ignored. In this embodiment, the original YOLOv8n network backbone is replaced with MobileNetV3 from CSPDarkNet53, with the addition of a Convolutional Block Attention Module (CBAM) for optimization. The C3Ghost module is used in the detection head to further reduce the number of parameters. This significantly improves detection efficiency while maintaining accuracy, reducing the computing power required on the deployment platform. The original YOLOv8n network consumes 8.2 GFLOPS, while the improved new network consumes 4.6 GFLOPS.

[0111] After acquiring the image data and radar data, data calibration is performed, specifically including spatial alignment and temporal alignment of the data.

[0112] In terms of spatial alignment, the yaw and pitch angles of the camera and radar are corrected to unify the image data and radar data into the same coordinate system, thus achieving spatial alignment of the data.

[0113] In terms of time alignment, the camera and radar are started synchronously through an external trigger signal, the data is sorted by timestamp, the time window is set based on the camera's timestamp, and the radar data with the smallest time difference and the image data are selected for time alignment; or vice versa, the time window is set based on the radar's timestamp and the image data with the smallest time difference are selected for time alignment.

[0114] The steps of correcting the yaw angle and pitch angle of the camera and radar specifically include:

[0115] Radar directly outputs radar yaw angle and radar elevation angle , and determine the camera yaw angle of the camera by the following formula And the camera pitch angle ;

[0116]

[0117]

[0118]

[0119] in,( ， ) is the center pixel coordinate of the target frame in the camera, 、 is the focal length of the camera, 、 is the optical center of the camera, ( ， ) is the coordinate of the target in the normalized plane;

[0120] The difference between the radar yaw angle and the camera yaw angle As a compensation value for correcting the radar yaw angle, the difference between the radar pitch angle and the camera pitch angle is As a compensation value for correcting the radar pitch angle, the specific operation is to directly add the corresponding compensation value to the radar yaw angle and radar pitch angle, thereby unifying the radar data into the camera coordinate system. Of course, in other embodiments, the image data can also be unified into the radar coordinate system.

[0121] It is easy to understand that there may be multiple different targets in the image data, so the camera pitch angle and camera yaw angle can be obtained for each target, and then the Therefore, as the compensation value for correcting the radar yaw angle, take The average value of is used as the compensation value for correcting the radar pitch angle.

[0122] The pre-trained fuzzy neural network model includes input layer, fuzzification layer, rule layer, normalization layer and output layer, such as Figure 4As shown in the figure, the input layer is used to input the visual feature vector and the radar feature vector into the fuzzification layer; the fuzzification layer is used to describe the input semantically through membership functions and map the input to fuzzy sets; the rule layer is used to reason and calculate the fuzzified input according to fuzzy rules; the normalization layer is used to convert the results of fuzzy reasoning and calculation into accurate fuzzy scores; and the output layer is used to output the degree of match between the image data and each target in the radar data. The number of structural layers of the fuzzy neural network is lower than that of deep learning frameworks such as convolutional neural networks. Therefore, its network structure is relatively simple, and the required computing power is only 3.2 MFLOPS, which is much lower than other algorithms. It is well suited for edge computing platforms with limited computing power, effectively reducing deployment costs and improving computing efficiency. To verify the effectiveness of this solution, this embodiment uses the method proposed in this solution and other existing algorithms (including four multimodal target detection fusion algorithms, namely AVOD, MV3D, F-PointNet and ContFuse) to conduct comparative tests on the visual module data and radar data collected in the same time period under the same experimental environment. Each round is repeated 100 times and the average fusion time is taken. A total of 56 rounds of experiments are conducted. The specific experimental comparison results are as follows: Figure 5 As shown, Figure 5 The FNN is the method proposed in this scheme. Figure 5 The horizontal axis corresponds to the number of running rounds, and the vertical axis corresponds to the average fusion time. Experimental data show that the time required for data fusion in this scheme is 0.18ms, which is 82% faster than other algorithms.

[0123] 1. Input layer

[0124] The image data obtained by the camera cannot be directly input into the fuzzy neural network, so it is necessary to process the image data through the YOLO algorithm to obtain the visual feature vector suitable for the fuzzy neural network input. x i .

[0125] in, , Indicates the image data goals, ( ， ) is the first The coordinates of the target, and are the width and height of the target bounding box, respectively. is the target category, is the confidence level of the target; after the radar data is preprocessed into point cloud data, the radar feature vector is formed :

[0126] ,in, Indicates the radar data goals, ( ， ) is the first The coordinates of the target, is the target distance detected by the radar, is the radar yaw angle, is the target speed detected by the radar, The target movement trend detected by the radar, including approach and distance.

[0127] 2. Fuzzy layer

[0128] 1) By Target and radar data The distance between the targets Describe and obtain three fuzzy sets, namely matching, partial matching, and non-matching.

[0129] in , described using a piecewise linear function:

[0130] .

[0131] 2) Target speed detected by radar Describe and obtain 5 fuzzy sets: low, slightly low, medium, slightly fast, fast, and use piecewise linear function to describe:

[0132] ,

[0133] ,

[0134] ,

[0135] ,

[0136] ;

[0137] 3) By analyzing the rate of change of the target bounding box in the image data Describe it and get two fuzzy sets: close and far, which are described using the following membership functions:

[0138] , .

[0139] 3. Rules layer

[0140] Fuzzification rules include:

[0141] Rule 1: If "Speed ​​= High" and "Radar Movement Trend = Approaching" and "Target Type = Car", then "Match Probability = High". The reasoning and calculation formula is:

[0142] .

[0143] Rule 2: If "Speed ​​= High" and "Radar Movement Trend = Approaching" and "Target Type = Electric Bicycle", then "Match Probability = High". The reasoning and calculation formula is:

[0144] .

[0145] Rule 3: If "Speed ​​= Slow" and "Radar Movement Trend = Far Away" and "Target Type = Electric Bicycle", then "Match Probability = High". The reasoning and calculation formula is:

[0146] .

[0147] Rule 4: If "Speed ​​= Small" and "Radar Movement Trend = Approaching" and "Target Type = Bicycle", then "Match Probability = High". The reasoning and calculation formula is:

[0148] .

[0149] Rule 5: If "Speed ​​= Medium" and "Radar Movement Trend = Far Away" and "Target Type = Bicycle", then "Match Probability = High". The reasoning and calculation formula is:

[0150] .

[0151] Rule 6: If "Speed ​​= High" and "Radar Movement Trend = Far Away" and "Target Type = Pedestrian", then "Match Probability = High". The reasoning and calculation formula is:

[0152] .

[0153] Rule 7: If "radar motion trend = close" and "visual detection motion trend = close", then "match probability = high". The reasoning and calculation formula is:

[0154] .

[0155] Rule 8: If "radar motion trend = far away" and "visual detection motion trend = far away", then "match probability = high". The reasoning and calculation formula is:

[0156] .

[0157] 4. Normalization layer

[0158] The normalization layer is implemented by the following formula:

[0159]

[0160] ,

[0161] ,

[0162] ,

[0163] in, 、 、 Distance fuzzy matching , speed fuzzy matching , motion trend fuzzy matching The activation weights of these three types of fuzzy matching results are: is the total activation weight. During model training, its initial value is set based on expert experience. The specific training process of the model is common knowledge and will not be described in detail in this embodiment.

[0164] 5. Output Layer

[0165] The output layer uses fuzzy scoring and a multi-target matching matrix to output a matching matrix between the image data and each target in the radar data. For N radar targets and M visual targets, an N×M matching matrix is ​​generated, and the Hungarian algorithm is used to extract the optimal match. In this example, there are two radar targets and three visual targets, and the FNN outputs a 2×3 matching matrix:

[0166]

[0167] For example, if the preset threshold is set to 0.8, when the matching degree between radar target 1 and visual target 1, and the matching degree between radar target 2 and visual target 3 both exceed 0.8, it indicates that radar target 1 matches visual target 1, and radar target 2 matches visual target 3. This means that radar target 1 and visual target 1 are identical and are detected true targets, radar target 2 and visual target 3 are identical and are detected true targets, and visual target 2 is a detection error and not a true target.

[0168] In summary, this solution uses fuzzy neural networks and weight distribution to fuse visual and radar data to achieve highly robust and low-cost target detection and recognition. It is suitable for cycling traffic and can solve the problems of low detection accuracy, high hardware cost, insufficient real-time performance, and lack of specific target classification for non-motor vehicles in complex environments caused by traditional fusion methods. It effectively improves the safety perception ability of cyclists in complex traffic scenarios and has the advantages of real-time and explainability.

Claims

1. A multi-target detection method, characterized in that: The following steps are involved: Step S1: Acquire image data and radar data, and perform data calibration; Step S2: obtaining a visual feature vector and a radar feature vector based on the image data and radar data after data calibration; Step S3, using a pre-trained fuzzy neural network model to perform fuzzy inference on the visual feature vector and the radar feature vector, and outputting a fuzzy inference result, wherein the fuzzy inference result includes a matching degree between the image data and each target in the radar data; Step S4: Filter out targets whose matching degree exceeds a preset threshold from the fuzzy inference results; The pre-trained fuzzy neural network model in step S3 includes an input layer, a fuzzification layer, a rule layer, a normalization layer and an output layer; The input layer is used to input the visual feature vector and the radar feature vector into the fuzzification layer; The fuzzification layer is used to semantically describe the input quantity through the membership function to map the input quantity to the fuzzy set; The rule layer is used to reason and calculate the fuzzified input according to the fuzzy rules; The normalization layer is used to convert the results of fuzzy reasoning and calculation into accurate fuzzy scores; The output layer is used to output the matching degree between the image data and each target in the radar data; The specific working process of the fuzzification layer is as follows: For the image data Target and radar data The distance between the targets Describe: ， ， The image data The coordinates of the target, ， The radar data The coordinates of the target; Three fuzzy sets are obtained: match, partial match, and mismatch, which are described using piecewise linear functions: ； Speed ​​of target detected by radar Describe and obtain 5 fuzzy sets: low, slightly low, medium, slightly fast, fast, and use piecewise linear function to describe: ， ， ， ， ； in, 、 、 、 、 are the piecewise linear functions corresponding to the five fuzzy sets of "low, slightly low, medium, slightly fast, and fast"; By calculating the rate of change of the target bounding box in the image data Describe it and get two fuzzy sets: close and far, which are described using the following membership functions: ， ， in, is the membership function corresponding to the "close" fuzzy set, is the membership function corresponding to the “away” fuzzy set.

2. A multi-target detection method according to claim 1, characterized in that: In step S1, the image data is acquired by a camera, and the radar data is acquired by a radar. The data calibration in step S1 includes spatial alignment and temporal alignment of the data. Specifically, the spatial alignment is performed by correcting the yaw angle and pitch angle of the camera and the radar, thereby unifying the image data and the radar data into the same coordinate system to achieve spatial alignment of the data. Time alignment involves synchronizing the camera and radar through an external trigger signal, sorting the data by timestamp, setting a time window based on the camera's timestamp, and selecting the radar data with the smallest time difference to align with the image data. Alternatively, you can use the radar timestamp as the reference, set the time window, and select the image data with the smallest time difference to align with the radar data.

3. A multi-target detection method according to claim 2, characterized in that: The specific process of correcting the yaw angle and pitch angle of the camera and radar is as follows: According to the radar yaw angle output by the radar and radar elevation angle , determine the camera yaw angle of the camera by the following formula And the camera pitch angle : in,( ， ) is the center pixel coordinate of the target frame in the camera, 、 is the focal length of the camera, 、 is the optical center of the camera, ( ， ) is the coordinate of the target in the normalized plane; Then the difference between the radar yaw angle and the camera yaw angle is calculated As a compensation value for correcting the radar yaw angle, the difference between the radar pitch angle and the camera pitch angle is As a compensation value for correcting the radar pitch angle, the image data and radar data are unified into the camera coordinate system; Or vice versa, the image data and radar data are unified into the radar coordinate system.

4. A multi-target detection method according to claim 1, characterized in that: The step S2 specifically processes the image data using the YOLO algorithm to obtain a visual feature vector suitable for fuzzy neural network input. ; And perform point cloud data preprocessing on radar data to obtain radar feature vector .

5. A multi-target detection method according to claim 4, characterized in that: The visual feature vector x i Specifically: ， in, Indicates the image data goals, ( ， ) is the first The coordinates of the target, and are the width and height of the target bounding box, respectively. is the target category, is the confidence level of the target; The radar characteristic vector y j Specifically: ， in, Indicates the radar data goals, ( ， ) is the first The coordinates of the target, is the target distance detected by the radar, is the radar yaw angle, is the target speed detected by the radar, It is the target movement trend detected by radar.

6. A multi-target detection method according to claim 1, characterized in that: The fuzzy rules include: Rule 1: If "Speed ​​= Fast" and "Radar Movement Trend = Approaching" and "Target Type = Car", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 1, Corresponding to "speed = fast", The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = car"; Rule 2: If "Speed ​​= Slightly Faster" and "Radar Movement Trend = Approaching" and "Target Type = Electric Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 2, The corresponding value is "speed = slightly faster", The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = electric bicycle"; Rule 3: If "Speed ​​= Low" and "Radar Movement Trend = Far Away" and "Target Type = Electric Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 3, The corresponding value is "speed = low". The corresponding value is "radar movement trend = far away". The corresponding value is "target type = electric bicycle"; Rule 4: If "Speed ​​= Slightly Low" and "Radar Movement Trend = Approaching" and "Target Type = Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 4, The corresponding value is "speed = slightly lower". The corresponding value is "radar movement trend = approaching". The corresponding value is "target type = bicycle"; Rule 5: If "Speed ​​= Medium" and "Radar Movement Trend = Far Away" and "Target Type = Bicycle", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 5, The corresponding speed is "medium". The corresponding value is "radar movement trend = far away". The corresponding value is "target type = bicycle"; Rule 6: If "Speed ​​= Fast" and "Radar Movement Trend = Far Away" and "Target Type = Pedestrian", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 6, Corresponding to "speed = fast", The corresponding value is "radar movement trend = far away". The corresponding value is "target type = pedestrian"; Rule 7: If "Radar Movement Trend = Approaching" and "Visual Detection Movement Trend = Approaching", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 7, The corresponding value is "Visual detection movement trend = approaching". Corresponding to "radar movement trend = approaching"; Rule 8: If "Radar Movement Trend = Far Away" and "Visual Detection Movement Trend = Far Away", then "Match Probability = High". The corresponding reasoning and calculation formula is: ； in, is the fuzzy score of rule 8, The corresponding value is "Visual detection movement trend = away". The corresponding value is "radar movement trend = far away".

7. A multi-target detection method according to claim 6, characterized in that: The fuzzy score output by the normalization layer is specifically: ， ， ， in, It is i The visual target and j The fuzzy score obtained by matching between radar targets, 、 、 Distance fuzzy matching , speed fuzzy matching , motion trend fuzzy matching The activation weights of these three types of fuzzy matching results are: is to select the maximum value among the fuzzy scores obtained from rules 1 to 6, The maximum value is selected from the fuzzy scores obtained from rule 7 and rule 8.

8. The multi-target detection method according to claim 1, wherein: The output layer is specifically based on the fuzzy score and the multi-target matching matrix, and outputs a matching matrix between the image data and each target in the radar data.

Images