Target detection method and device based on multiple sensors and storage medium

Abstract

The invention discloses a multi-sensor-based target detection method and device and a storage medium. The multi-sensor-based target detection method comprises the steps of obtaining current visual data and current radar data in the same scene; the current visual data and the current radar data are matched in a radar coordinate system where the radar data are located, first matched data and first unmatched data are obtained, and the first unmatched data comprise the current unmatched visual data and the current unmatched radar data; performing matching processing on the current unmatched visual data and the current unmatched radar data in an image coordinate system where the visual data are located to obtain second matching data; and determining a current target detection result according to the first matching data and the second matching data. According to the scheme, the accuracy of target detection can be improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method, device and storage medium based on multiple sensors. Background Technology

[0002] Object detection technology has always been an important research direction in the fields of computer vision and artificial intelligence, aiming to identify target objects in the current scene and determine their relevant information in images. Object detection technology is now widely used in fields such as autonomous driving and security detection; for example, it can determine the driving status of vehicles on the road through video detection.

[0003] Traditional methods primarily rely on terminal equipment such as visible light cameras to acquire video images for target detection. While visible light cameras can obtain category information of the target object, they are not easy to obtain motion information. Therefore, to improve target detection accuracy, current research attempts combine visible light cameras with radar, which is better at detecting motion information, thus fusing radar and vision dimensions to achieve target detection.

[0004] However, due to differences in hardware operating principles, the current visual data from a camera and the point cloud data from a radar are not easily combined. For example, cameras are not suitable for detecting distant targets, and radar is not suitable for detecting stationary targets. This means that distant targets may not be detected by the camera, and stationary targets may not be detected by the radar. Even if a target object with both visual and point cloud data is detected, the coordinate transformation and mapping processes between the visual and point cloud data may cause significant errors between the two sets of data, affecting the accuracy of target detection and leading to detection errors. Summary of the Invention

[0005] This application provides at least one target detection method, apparatus, device, and computer-readable storage medium based on multiple sensors.

[0006] The first aspect of this application provides a target detection method based on multiple sensors, comprising: acquiring current visual data and current radar data in the same scene; performing matching processing on the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data, wherein the first unmatched data includes currently unmatched visual data and currently unmatched radar data; performing matching processing on the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located to obtain second matched data; and determining a current target detection result based on the first matched data and the second matched data.

[0007] In one embodiment, the step of matching the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located to obtain second matching data includes: obtaining a target mapping model; generating a current radar detection box corresponding to the currently unmatched radar data in the image coordinate system according to the target mapping model; and matching the current radar detection box with the currently unmatched visual detection box of the currently unmatched visual data to obtain the second matching data.

[0008] In one embodiment, obtaining the target mapping model includes: obtaining current matching visual data and current matching radar data from the first matching data, wherein the current matching visual data includes the size information of the current matching visual detection box and the current matching radar data includes radar distance information; and performing fitting processing based on the size information and the radar distance information to obtain the target mapping model.

[0009] In one embodiment, the step of matching the current radar detection box with the current unmatched visual detection boxes of the current unmatched visual data to obtain the second matching data includes: determining the degree of association between each radar detection box and each current unmatched visual detection box based on the cross-union ratio between each radar detection box and each current unmatched visual detection box; and determining the second matching data from each current unmatched visual data and each current unmatched radar data based on the degree of association.

[0010] In one embodiment, the step of matching the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data includes: transforming the current visual position of the current visual data to the radar coordinate system to obtain a visual transformation position; determining the degree of correlation between each current visual data and each current radar data based on the distance between the visual transformation position and the current radar position of the current radar data; and determining the first matched data and the first unmatched data from each current visual data and each current radar data based on the degree of correlation.

[0011] In one embodiment, determining the first matching data and the first unmatched data from each current visual data and each current radar data based on the degree of correlation includes: constructing an association matrix between each current visual data and each current radar data based on the degree of correlation between them; determining the first matching data based on the association matrix and a preset minimum weight matching algorithm; and determining the data in each current visual data and each current radar data other than the first matching data as the first unmatched data.

[0012] In one embodiment, the current target detection result includes a current target object. After determining the current target detection result based on the first matching data and the second matching data, the method further includes: acquiring historical target detection results detected in historical time periods, wherein the historical time periods are earlier than the current time period corresponding to the current target detection result, and the historical target detection results include historical target objects; predicting each predicted radar position corresponding to each historical target object in the current time period; matching each historical target object with each current target object based on each predicted radar position and each current radar position to obtain a first matched object and a first unmatched object, wherein the first unmatched object includes a first unmatched historical object and a first unmatched current object; predicting each predicted visual position corresponding to the first unmatched historical object in the current time period; matching each first unmatched historical object with each first unmatched current object based on each predicted visual position and each current visual position to obtain a second matched object; and determining a continuous detection result based on the first matched object and the second matched object.

[0013] In one embodiment, predicting the predicted radar positions corresponding to each historical target object in the current time period includes: acquiring radar sensor errors and visual sensor errors; determining a covariance matrix and a measurement noise matrix based on the radar sensor errors and the visual sensor errors; determining an observation matrix based on the data source of the historical target objects; constructing a Kalman prediction model based on the covariance matrix, the measurement noise matrix, the observation matrix, and the acquired process noise matrix; and inputting the historical radar positions into the Kalman prediction model to obtain the predicted radar positions output by the Kalman prediction model.

[0014] In one embodiment, determining the covariance matrix and the measurement noise matrix based on the radar sensor error and the visual sensor error includes: comparing the radar sensor error with the visual sensor error; determining matrix elements in the covariance matrix and the measurement noise matrix based on the radar sensor error if the radar sensor error is less than the visual sensor error; and determining matrix elements in the covariance matrix and the measurement noise matrix based on the visual sensor error if the radar sensor error is greater than or equal to the visual sensor error.

[0015] In one embodiment, after determining the continuous detection result based on the first matching object and the second matching object, the method further includes: obtaining unmatched historical objects other than the first matching object and the second matching object from each historical target object; obtaining the number of matching failures of the unmatched historical objects; deleting the unmatched historical objects in response to the number of matching failures being greater than a preset number threshold; and accumulating the number of matching failures of the unmatched historical objects in response to the number of matching failures being less than or equal to the preset number threshold.

[0016] A second aspect of this application provides a multi-sensor-based target detection device, comprising: an acquisition module for acquiring current visual data and current radar data in the same scene; a first matching module for matching the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data, wherein the first unmatched data includes currently unmatched visual data and currently unmatched radar data; a second matching module for matching the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located to obtain second matched data; and a result determination module for determining a current target detection result based on the first matched data and the second matched data.

[0017] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-described multi-sensor-based target detection method.

[0018] The fourth aspect of this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a processor, implement the above-described multi-sensor-based target detection method.

[0019] The above scheme, by acquiring and analyzing the current visual data collected by the camera and the current radar data collected by the radar in the same scene at the current time period, can compensate for the reduced target detection accuracy caused by the limited precision of a single sensor. Then, the same object represented by the two types of data can be determined through radar-visual matching. Specifically, the current visual data and current radar data are matched in the radar coordinate system where the radar data resides, resulting in first matched data (successfully matched) and first unmatched data (unmatched), including both currently unmatched visual and radar data. The unmatched visual and radar data are then matched in the image coordinate system where the visual data resides, resulting in second matched data (successfully matched) at the visual data dimension. The target detection result is determined based on the first and second matched data. Therefore, this application can achieve comparative correlation between radar and visual data at their respective scales through a two-stage matching method, improving the accuracy of target detection. Even if a single sensor malfunctions, correlation will still occur within the coordinate system of the normally functioning sensor, without affecting the detection accuracy.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0022] Figure 1 This is an exemplary target detection diagram in the multi-sensor-based target detection method of this application;

[0023] Figure 2 This is a flowchart illustrating an exemplary embodiment of the multi-sensor-based target detection method of this application;

[0024] Figure 3 This is another exemplary target detection diagram in the multi-sensor-based target detection method of this application;

[0025] Figure 4 This is an exemplary continuous detection flowchart of the multi-sensor-based target detection method of this application;

[0026] Figure 5 This is an exemplary flowchart illustrating the combination of single-frame detection and continuous detection in the multi-sensor-based target detection method of this application.

[0027] Figure 6This is an exemplary summary flowchart of the multi-sensor-based target detection method of this application;

[0028] Figure 7 This is a block diagram illustrating a multi-sensor-based target detection device in an exemplary embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application;

[0030] Figure 9 This is a schematic diagram of the structure of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0031] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0032] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0033] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this document means two or more. Moreover, the term "at least one" in this document means any combination of at least two of any one or more of a plurality of objects. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0034] To facilitate understanding, one of the applicable scenarios of this application will be illustrated by example.

[0035] Traditional target detection methods primarily rely on terminal equipment such as visible light cameras to acquire video images for target detection. While visible light cameras can obtain category information of the target object, they are not easy to obtain motion information. Therefore, to improve target detection accuracy, current research attempts combine visible light cameras with radar, which is better at detecting motion information, thus fusing radar and vision dimensions to achieve target detection.

[0036] While both radar sensors and vision sensors (image sensors) can detect targets, their applicable scenarios are vastly different. For example, millimeter-wave radar actively emits electromagnetic waves and receives signals of the same frequency. It has a very high detection probability for moving objects or objects with a large RCS (radar cross-section). Its advantages lie in its ability to accurately acquire spatial and motion information of targets, detect targets at a distance from the radar sensor, and is less affected by environmental factors such as lighting. However, radar sensors have disadvantages in that they cannot obtain high-accuracy target identification information (such as category information), and cannot accurately detect all targets when the target is stationary (e.g., a low detection probability for stationary objects). Visual data such as video / images, on the other hand, can obtain high-accuracy target identification information and can accurately detect all targets at close to medium ranges. However, vision sensors have disadvantages in that they are not easy to accurately acquire motion and spatial information of targets, have poor detection performance for targets at a distance from the vision sensor, and are easily affected by environmental factors such as poor lighting and inclement weather (e.g., lower target identification accuracy at long distances, at night, or in rainy or foggy conditions).

[0037] Even if a target object with both visual and point cloud data is detected, significant errors may occur due to coordinate transformation and mapping processes between the two datasets, affecting target detection accuracy and causing detection errors. Therefore, due to differences in hardware operating principles, it is not easy to combine camera visual data and radar point cloud data.

[0038] For example, you can refer to... Figure 1 As shown, Figure 1 This is an exemplary target detection diagram from the multi-sensor-based target detection method of this application. This application primarily uses a vehicle as the target object for illustration. Figure 1 The left-middle image shows the detection scene from the previous frame. Figure 1 The right image shows the detection scene in the next frame (or the current frame) (i.e., after a period of time, the target object moves from the state shown in the left image to the state shown in the right image). The green boxes are visual detection boxes, and the red dots are the positions detected by radar.

[0039] In the scene, A, B, C, and D represent four visual targets, and ①②③④ represent radar targets mapped to the image after calibration. If a visual target has a corresponding green detection box, it means that the target has been detected by the visual sensor; otherwise, although the target exists, it may not be detected by the visual sensor due to its distance. In the left image, since the vehicle is moving, all targets can usually be detected by the radar, while the visual sensor can only detect targets B and C at close range. After a period of time, the status of all targets is as shown in the right image. Due to some targets being stationary, target congestion, and / or targets being too close, the radar sensor cannot detect target C, and due to the distance being too far, the visual sensor cannot detect target A.

[0040] For example, radar-visual integrated machines typically have a maximum detection range of 500 meters, but the vision sensor within it may not be able to accurately detect the visual data of a target object at 200 meters. Furthermore, due to the perspective effect of objects appearing larger when closer and smaller when farther away, the actual physical distance corresponding to each pixel increases with distance during visual detection. If the target object is at 200 meters, a one-pixel jitter in the visual detection box could result in a distance jitter exceeding 20 meters after mapping to the BEV (Bird's Eye View) coordinate system. This positional mapping error not only affects the target association accuracy of radar and vision but also severely impacts the target association accuracy between consecutive frames during continuous detection.

[0041] Therefore, the multi-sensor-based target detection method proposed in this application can use radar and vision sensors to detect target objects based on the radar-vision matching method. This allows for the complementary advantages of radar and vision solutions, expanding the detection scenarios for target objects. By effectively fusing radar and vision data using appropriate methods, the device can operate in more detection scenarios while obtaining more detailed and accurate identification and detection information for target objects.

[0042] It should be noted that the radar data (measurement information of the target object) that radar sensors can acquire includes, but is not limited to, distance, angle, radial speed, and radar cross-section (RCS). Normal millimeter-wave radar detects targets as a clustered region. However, for decision-level target fusion, radar targets are usually just clustered points, while visual targets are typically rectangular regions in the image (such as rectangular detection boxes). During target localization, although visual targets cannot directly obtain the distance information of the target object, after calibrating the radar coordinate system and the image coordinate system, the mapped position of the visual target in the radar coordinate system (i.e., the distance between the sensor and the target object) can be obtained. Under normal calibration conditions, the radar measurement position can only accurately describe the overall position of a target (e.g., the geometric center of a car), while in close, unobstructed proximity, the visual mapped position can describe in detail the distances of specific components within a target (e.g., the front, rear, and license plate of a car).

[0043] Therefore, in the specific implementation of target detection methods, such as in detection scenarios like parks, roads, bridges, industrial parks, and squares, vision-based target detection methods can obtain the type of the target object, while radar-based target detection methods can obtain the spatial location and velocity of the target object. By correctly matching / fusing radar and visual targets, and based on this correct fusion, data services can be provided for traffic flow indicators (such as vehicle flow, space occupancy, time occupancy, queue length, etc.) in the detection scenario.

[0044] Please see Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the multi-sensor-based target detection method of this application. Specifically, it may include the following steps:

[0045] Step S110: Obtain the current visual data and current radar data in the same scene.

[0046] Here, current visual data refers to data obtained by analyzing video and / or image data acquired by a visual sensor (image sensor), while current radar data refers to data obtained by analyzing data acquired by a radar sensor. For ease of understanding, one piece of visual data can be equivalent to a visual object detected by a visual sensor, and one piece of radar data can be equivalent to a radar object detected by a radar sensor. The visual sensor and radar sensor can be integrated into the same data acquisition device (e.g., a radar-visual integrated machine), or they can be installed on separate data acquisition devices; this is not limited here.

[0047] It should be noted that the current visual data and current radar data are obtained by the visual sensor and radar sensor respectively from collecting data on the same scene. The fields of view of the visual sensor and radar sensor can be completely or partially the same, which is not limited here. The same field of view means the same scene. For example, if the visual sensor detects two regions A and B, and the radar sensor detects two regions B and C, then B is the same field of view (the same scene).

[0048] It should also be noted that the acquired visual and radar data can be time-series aligned, which is a common data preprocessing method in the field of radar-visual matching, and will not be elaborated here.

[0049] For example, reference can be made to, such as Figure 3 As shown, Figure 3 This is another exemplary target detection diagram in the multi-sensor-based target detection method of this application. Figure 3 The left side represents the camera detection view, and the right side represents the radar detection view. Based on the visual detection algorithm and radar detection algorithm in this field, the number and location of target objects in the current scene can be detected respectively (excluding situations where target objects are outside the sensor's detection field of view or are occluded). The specific visual detection algorithm and radar detection algorithm used in this application are not limited; they can be set as needed by referring to relevant available algorithms in this field, and will not be elaborated here. The visual detection algorithm can provide the position of the visual detection box of the target object in real time, thereby establishing a target sequence and its corresponding target box sequence as one of the current visual data. For example, the current visual data may include:

[0050]

[0051] in, BoxPt represents the target object ID determined during the visual inspection process. i This indicates the position of the visual inspection box for each target object determined during the visual inspection process.

[0052]

[0053] in, These represent the pixel coordinates of the top-left, top-right, bottom-left, and bottom-right vertices of the visual detection box, respectively.

[0054] For example, input visual detection box BoxPt iThen, the center point (U, V) of the visual detection box is obtained. Finally, (U, V) is converted into the corresponding output (X, Y) using a pre-calibrated mapping algorithm, which is the actual position of the visual target in the world coordinate system. There are many pixel and 2D position calibration methods, commonly used methods include camera intrinsic and extrinsic parameter calibration methods and four-point calibration methods, which will not be elaborated here. The main purpose is to obtain the visual detection box BoxPt. i This corresponds to the actual position (X,Y) in the world coordinate system.

[0055] Similarly, radar can also detect the number of target objects and point cloud data, establishing the following target sequence and corresponding point cloud set sequence as one of the current radar data, such as the current radar data including:

[0056] Rtrg1,CartPtSet1

[0057] Rtrg2,CartPtSet2

[0058] Rtrg3,CartPtSet3

[0059] in, CartPtSet represents the target object ID determined during radar detection. i This represents the point cloud set of each target object identified during radar detection.

[0060] The above data can be considered as measurement data of the target object by visual sensors and radar sensors. All measurement data can be processed using algorithms such as clustering and calibration to construct the following mapping relationship:

[0061] Rloc i =h cluster (CartPtSet i )

[0062] Vloc i =h calib (BoxPt i )

[0063] Among them, Rloc i h represents the position of a radar target in its world coordinate system (radar position). cluster (*) indicates a clustering algorithm that performs clustering processing on radar point cloud data of the target object. Vloc i This indicates the position of a visual target in its world coordinate system (visual position). calib (*) represents a calibration function that converts the pixel points of the visual detection box of the target object into their positions in the world coordinate system.

[0064] For example, the clustering methods used can include, but are not limited to, density-based spatial clustering of applications with noise (DBSCAN). DBSCAN is a density-based spatial clustering algorithm and is one of the most widely used clustering methods in radar algorithms, which will not be elaborated on here. Its purpose is to obtain the number and location of clustered target objects based on radar data (point cloud data). That is, in a vehicle detection scenario, after inputting radar point cloud data, the corresponding number and location of vehicles can be obtained.

[0065] Therefore, current visual data can also include the current visual location, and current radar data can also include the current radar location. Based on the radar data and visual data processed in the aforementioned embodiments, subsequent matching processing can be performed using a matching algorithm to attempt to associate and fuse radar data and visual data of the same target object (fusion of radar object and visual object), and the following mapping relationship can be constructed:

[0066]

[0067] in, This indicates the location of the fusion target (target object) after data matching. This indicates the speed at which the target (object) is fused after data matching. f(*) represents the process of matching and associating radar targets and visual targets to obtain the fused target, which is the target detection result of the object. How to determine and implement f(*) is the key data matching and fusion process in this application.

[0068] Step S120: Match the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain the first matched data and the first unmatched data. The first unmatched data includes the currently unmatched visual data and the currently unmatched radar data.

[0069] In conjunction with the steps described above, when matching and associating the current visual data and the current radar data in the radar coordinate system, the position of the current visual data can be mapped to the radar coordinate system. This allows for association with the radar data based on positional relationships, resulting in the first matched data and / or the first unmatched data.

[0070] The location of the current visual data can be determined based on the visual detection bounding box of the current visual data. This can be achieved, for example, by using, but not limited to, the coordinates of the visual detection bounding box (e.g., one of the coordinates of the four vertices, or the center coordinates of the detection bounding box calculated from the four vertices, etc., without limitation here). The coordinate mapping process can be determined using a preset coordinate mapping function or a real-time acquired coordinate mapping function, which will not be elaborated here. The location of the current radar data can be determined, for example, by using, but not limited to, the location of the cluster center of the radar point cloud.

[0071] The first matching data includes successfully matched visual data and matched radar data. Mutually matched visual data and radar data can form a radar-visual matching pair, which can represent a successfully associated fused target (target object). The first unmatched data includes currently unmatched visual data and currently unmatched radar data that failed to match.

[0072] Step S130: Match the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located to obtain the second matched data.

[0073] The preceding steps will be explained in conjunction with the steps described above. After the matching process, first matched data and / or first unmatched data may be obtained. This explanation will focus on a scenario where both first matched data and first unmatched data exist. If matching fails in the radar coordinate system, the process can be switched to the image coordinate system. This application performs association operations in both coordinate systems, ensuring that even if a single sensor malfunctions, association will still occur within the coordinate system of the normally functioning sensor, thus maintaining the target detection accuracy.

[0074] Therefore, in this step, the position of the current unmatched radar data can be mapped to the image coordinate system, and then associated with the current unmatched visual data based on the positional relationship to obtain the second matched data and / or the second unmatched data.

[0075] The method of mapping the position of the current unmatched radar data to the image coordinate system can be similarly referred to the coordinate mapping method in the previous steps; or, it can be based on a pre-built mapping model to map the current unmatched radar data to a radar detection box in the image coordinate system, and then determine the position of the current unmatched radar data in the image coordinate system based on such as the vertex coordinates of the radar detection box.

[0076] Step S140: Determine the current target detection result based on the first matching data and the second matching data.

[0077] To illustrate the steps outlined above, the first and second matching data are successfully matched radar-visual data from the current visual and radar data. These successfully matched radar-visual data are fused into a single radar-visual data pair, each representing a target object. Therefore, the current target detection result can be determined based on the first and second matching data. The current target detection result can include the detected target objects, which may include visual information (such as category information) and radar information (such as motion information).

[0078] Optionally, the second unmatched data can also be used as one of the current target detection results. These second unmatched data may include missed detections, false detections, and / or data that appears for the first time in the acquisition field of view. Therefore, the second unmatched data can also be recorded as the current target detection result, and the decision on whether to apply its data information can be made in subsequent processes.

[0079] As can be seen, this application analyzes the current visual data collected by the camera and the current radar data collected by the radar in the same scene at the current time period. This compensates for the reduced target detection accuracy caused by the limited precision of a single sensor. Then, the same object represented by the two types of data can be determined through radar-visual matching. Specifically, the current visual data and the current radar data are matched in the radar coordinate system where the radar data is located, resulting in first matched data and first unmatched data that are successfully matched in the radar data dimension. The first unmatched data includes currently unmatched visual data and currently unmatched radar data. The currently unmatched visual data and the currently unmatched radar data are matched in the image coordinate system where the visual data is located, resulting in second matched data that are successfully matched in the visual data dimension. The target detection result is determined based on the first and second matched data. Thus, this application can achieve comparative correlation between radar data and visual data at their respective scales through a two-stage matching method, improving the accuracy of target detection. Even if a single sensor fails, correlation will still be performed in the coordinate system corresponding to the normally functioning sensor, without affecting the change in detection accuracy.

[0080] Based on the above embodiments, this application embodiment describes the steps of matching currently unmatched visual data and currently unmatched radar data in the image coordinate system where the visual data is located to obtain second matched data. Specifically, the method of this embodiment includes the following steps:

[0081] Obtain the target mapping model; generate the current radar detection box corresponding to the current unmatched radar data in the image coordinate system based on the target mapping model; perform matching processing between the current radar detection box and the current unmatched visual detection box of the current unmatched visual data to obtain the second matching data.

[0082] In conjunction with the foregoing embodiments, this embodiment mainly describes the matching process of the image coordinate system. The target mapping model can be preset or acquired when needed; no limitation is made here. The target mapping model is used to map radar data to the image coordinate system, for example, generating the corresponding current radar detection box in the image coordinate system based on the current radar data.

[0083] Then, the current radar detection box can be matched with the current unmatched visual detection box of the current unmatched visual data to obtain the second matched data and the second unmatched data.

[0084] Based on the above embodiments, this application embodiment describes the steps for obtaining a target mapping model. Specifically, the method of this embodiment includes the following steps:

[0085] Obtain the current matching visual data and the current matching radar data from the first matching data. The current matching visual data includes the size information of the current matching visual detection box, and the current matching radar data includes the radar distance information. Perform fitting processing based on the size information and radar distance information to obtain the target mapping model.

[0086] Based on the foregoing embodiments, this embodiment describes a method for obtaining a target mapping model.

[0087] For example, the target mapping model can be temporarily determined based on the first matching data during each application, or it can be predetermined and stored in a storage medium. It is understood that when performing association in the image coordinate system, if the matching association method used is IOU comparison, but since the current radar position represented by the radar data only has one cluster point, this application needs to obtain a target mapping model between the current radar data and the detection box, and generate the current radar detection box corresponding to the current radar data in the image coordinate system. For example, inputting the current radar position (X,Y) of the current radar data, the detection box information of this radar target can be directly generated through the target mapping model, such as...

[0088] Specifically, considering the perspective characteristics of the target detection scene, the farther the target object is from the data acquisition device, the smaller the target object appears in the acquisition field of view, which conforms to a certain linear relationship. Therefore, an initial linear mapping model between the target distance (X,Y) and the target width and height (w,h) is constructed:

[0089] w = k w y -1 +b w

[0090] h = k h y -1 +b h

[0091] This involves solving for k and b in the linear mapping model through data fitting. Therefore, during model fitting, successfully matched and associated fusion targets (target objects) at different distances can be selected from the aforementioned process; this is the first matching data. Based on the current matching visual data (size information of the current matching visual detection box) and current matching radar data (radar distance information, i.e., the distance between the radar and the target object) of several target objects in the first matching data, these are substituted into the initial linear mapping model to solve for the parameters, obtaining the parameter k of the linear model. w ,b w ,k h ,b h This yields a usable linear mapping model.

[0092] Furthermore, in specific applications, by substituting any acquired radar data (radar distance information), the width and height information of the detection box (the size information of the current radar detection box) at the corresponding position in the image coordinate system can be calculated using a linear mapping model. Then, by calling a pre-determined coordinate mapping function, the current radar position of the current radar data is projected onto the image coordinate system. The projected current radar position can be used as the lower edge center point (U, V) of the current radar detection box. Combining the center point (U, V) with the width and height (w, h) of the current radar detection box represents the current radar detection box. This allows for the determination of the current radar detection box in the image coordinate system where the radar object should appear, based on the radar distance information from the current radar data.

[0093] It should be noted that the fitting process of this linear mapping model can be run only after the radar-visual matching association is completed, and after the fitting is completed, it is not necessary to continue using the successfully associated data. If all the current visual data and current radar data obtained in the specific implementation process are successfully matched (i.e., there is only the first matching data and no first unmatched data), the current target detection result can be directly determined based on the first matching data. Therefore, it is also possible to choose not to implement the process of constructing the target mapping model in this embodiment.

[0094] Based on the above embodiments, this application embodiment describes the step of matching the current radar detection box with the current unmatched visual detection box of the current unmatched visual data to obtain second matched data. Specifically, the method of this embodiment includes the following steps:

[0095] Based on the cross-union ratio between each radar detection box and each currently unmatched visual detection box, the degree of association between each radar detection box and each currently unmatched visual detection box is determined; based on the degree of association, second matching data is determined from each currently unmatched visual data and each currently unmatched radar data.

[0096] Referring to the foregoing embodiments, after obtaining the radar detection boxes corresponding to each radar data, the Intersection over Union (IOU) between each radar detection box and each currently unmatched visual detection box can be calculated in the image coordinate system to construct the association matrix between each radar detection box and each currently unmatched visual detection box. The degree of association in the association matrix is ​​then determined based on the IOU between each radar detection box and each currently unmatched visual detection box.

[0097] For example, in a radar detection box where the radar target has been converted to an image coordinate system. Subsequently, the mathematical expression of the IOU distance model between radar target i and visual target j in the association method of this embodiment can be:

[0098]

[0099] Here, intersection represents the intersection of a radar detection box and a currently unmatched visual detection box, and union represents the union of these two detection boxes.

[0100] intersection=max(u2-u1+1.0,0)*max(v2-v1+1.0,0)

[0101] union = S A +S B -intersection

[0102] S A =(u a2 -u a1 +1.0)*(v a2 -v a1 +1.0)

[0103] S B =(u b2 -u b1 +1.0)*(v b2 -v b1 +1.0)

[0104] Where S A and S B These are the areas of the two detection boxes (box a and box b), where (u1, v1) is the coordinate of the top-left corner of the detection box, and (u2, v2) is the coordinate of the bottom-right corner of the detection box.

[0105] Then, the least weighted matching algorithm (such as the KM algorithm or the Hungarian algorithm) can be used to solve the correlation matrix to obtain the second matching data that is successfully matched and correlated among each currently unmatched visual data and each currently unmatched radar data.

[0106] Based on the above embodiments, this application embodiment describes the steps of matching current visual data and current radar data in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data. Specifically, the method of this embodiment includes the following steps:

[0107] The current visual position of the current visual data is transformed into the radar coordinate system to obtain the visual transformation position; the degree of correlation between each current visual data and each current radar data is determined based on the distance between the visual transformation position and the current radar position of the current radar data; and the first matching data and the first unmatched data are determined from each current visual data and each current radar data based on the degree of correlation.

[0108] The foregoing embodiments primarily illustrate a method for matching and associating elements based on the degree of correlation (IOU) in an image coordinate system. This embodiment primarily illustrates a method for matching and associating elements based on the degree of correlation, such as Euclidean distance, in a radar coordinate system.

[0109] For example, the current visual position of the current visual data (such as the coordinates of the center point of the visual detection box) can be transformed into the radar coordinate system using a pre-calibrated mapping function to obtain the visually transformed position. Then, the distance between radar target i and visual target j in the radar coordinate system can be determined using the Euclidean distance calculation method, the mathematical expression of which is:

[0110]

[0111] Among them, (x i ,y i (x) represents the radar target location. j ,y j () indicates the location of the visual target.

[0112] Therefore, an association matrix can be constructed between each radar target (current radar data) and each visual target (current visual data). The Euclidean distance between each current radar data and each current visual data can be calculated and used as the degree of association between each pair of data in the association matrix, Wij. To prevent misassociation between distant targets, a maximum distance threshold can be set for suppression. If the Euclidean distance between two targets exceeds the maximum distance threshold, Wij can be replaced by a preset maximum value. The logical expression is as follows:

[0113] if Wij>d max then Wij = W max

[0114] It should be noted that, in the matching process where IOU distance is used as the degree of association, a corresponding maximum distance threshold can also be set, and maximum value replacement processing can be implemented. This will not be elaborated here.

[0115] Based on the degree of correlation in the correlation matrix, the correlation matrix can be solved using a minimum weight matching algorithm (such as the KM algorithm) to obtain the first matching data and the first unmatched data between each current visual data and each current radar data.

[0116] It should be noted that the correlation matrix is ​​mainly used to describe whether a radar target and any visual target are the same target object. For example, the following correlation matrix can be established, where the row data represents radar targets and the column data represents visual targets.

[0117]

[0118] In this correlation matrix, Wij represents the correlation degree between the i-th radar target and the j-th visual target in the current frame. In different matching processes, the correlation degree can be calculated using the Euclidean distance between two targets in the radar coordinate system or the IOU distance between two detection boxes in the image coordinate system, as mentioned in the previous embodiments. In the preceding steps, the radar target and the visual target have been transformed into a point (X,Y) in the radar coordinate system (or world coordinate system). Therefore, in the process of determining the first matching data (referred to as the first matching process), the calculated Euclidean distance can be used to determine the correlation degree of the first matching process. Furthermore, the correlation matrix shown in this embodiment can also be similarly applied to the process of determining the second matching data (referred to as the second matching process) in the previous embodiments; that is, the calculated IOU distance can be used to determine the correlation degree of the second matching process.

[0119] After the association matrix is ​​constructed, for any matching result in the association matrix, a weighted sum can be obtained, for example... and Related, and Related, and Related, and When associated, the sum of their weights is:

[0120] W num =W 11 +W 22 +W 33 +W 44

[0121] It is understandable that the closer a radar target and a visual target are, the more likely they are to belong to the same target object; that is, when W... numWhen the minimum weight is reached, the current association is considered the best matching result. Therefore, the association matrix can be solved using the KM algorithm, and the matching result corresponding to the globally minimum weight is the best matching result (first matching data and / or second matching data). After solving the association matrix for both matching processes, the following matching results are typically obtained: ① successfully associated radar targets and visual targets; ② unassociated radar targets; ③ unassociated visual targets, equivalent to the first and second matching data (successfully associated radar targets and visual targets) and the second unmatched data (unassociated radar targets and unassociated visual targets). All three types of data can be selected as the current target detection result, and these three types of targets can be treated as independent fusion target objects for subsequent continuous detection.

[0122] Based on the above embodiments, this application embodiment describes the steps of determining first matching data and first unmatched data from each current visual data and each current radar data according to the degree of correlation. Specifically, the method of this embodiment includes the following steps:

[0123] Based on the degree of correlation between each current visual data and each current radar data, an association matrix is ​​constructed between each current visual data and each current radar data; the first matching data is determined based on the association matrix and a preset minimum weight matching algorithm; and the data in each current visual data and each current radar data other than the first matching data are determined as the first unmatched data.

[0124] Referring to the foregoing embodiments, after obtaining the correlation degree Wij between each current visual data and each current radar data, a correlation matrix between each current visual data and each current radar data is constructed. The minimum matching weight (i.e., within W) of various combinations in the correlation matrix is ​​solved using a preset minimum weight matching algorithm. num When the minimum value is reached, the optimal combination can be considered obtained, which is the first matching data.

[0125] In each current visual data and each current radar data, apart from the first matching data, the remaining data is determined as the first unmatched data. Subsequently, the first unmatched data can be subjected to a second matching process through the image coordinate system (refer to the description of the aforementioned embodiment for details), which will not be elaborated here.

[0126] Based on the above embodiments, this application embodiment describes the steps after determining the current target detection result according to the first matching data and the second matching data. The current target detection result includes the current target object. Specifically, the method of this embodiment includes the following steps:

[0127] The system acquires historical target detection results from historical time periods, which are earlier than the current time period corresponding to the current target detection result. These historical target detection results include historical target objects. It then predicts the predicted radar positions of each historical target object in the current time period. Based on the predicted radar positions and the current radar positions, it matches each historical target object with each current target object to obtain a first matched object and a first unmatched object. The first unmatched object includes first unmatched historical objects and first unmatched current objects. Finally, it predicts the predicted visual positions of the first unmatched historical objects in the current time period. Based on the predicted visual positions and the current visual positions, it matches each first unmatched historical object with each first unmatched current object to obtain a second matched object. Finally, it determines the continuous detection result based on the first and second matched objects.

[0128] The preceding embodiments will be used for illustration, which mainly describe the process of target detection in the current time period (current frame). In this application, in addition to target detection based on current radar data and current visual data, historical radar data and historical visual data acquired from historical time periods (historical frames) can also be combined to achieve continuous target detection. The historical time periods are earlier than the current time period.

[0129] It should be noted that, typically, after completing target detection on a frame of data, the target detection result for that frame can be obtained. This result can include the detected target objects. For example, after completing target detection on the current frame, the current target detection result (including the current target objects) can be obtained, such as ① successfully associated radar targets and visual targets mentioned in the previous embodiments; ② unassociated radar targets; ③ unassociated visual targets. Similarly, after completing target detection on historical frames, historical target detection results (including historical target objects) can also be obtained.

[0130] In the aforementioned embodiments, the radar-view fusion target was determined using a radar-view matching method. However, in applications requiring continuous target detection, the fusion target detected in a single frame is not assigned a fusion ID; it typically only contains information such as category, location, and velocity. Therefore, this embodiment employs a continuous detection algorithm to assign a fusion ID to the fusion target after radar-view matching, enabling continuous detection of targets in the current detection scenario. The mathematical expression of this mapping relationship can be as follows:

[0131]

[0132] in, This indicates the ID of the target to be fused after continuous detection. This involves fusing target information such as ID, position, and velocity from historical frames (e.g., the frame preceding the current frame). g( *) represents the continuous detection algorithm. The main purpose of the continuous detection algorithm is to match and associate the fusion target of the current frame with the fusion target of the previous frame, and complete the assignment of the fusion ID of the current frame. How to determine and implement g( * The process of detection is the key continuous detection process after radar-visual matching and target fusion in this application.

[0133] It should be noted that each target participating in continuous monitoring should possess complete target attributes, such as:

[0134] S F ={x,y,v x ,v y ,l,u,r,d,source}

[0135] Among them, a certain fusion target S F In the attribute information, l, u, r, d represent the coordinates of the upper left and lower right corners of the visual detection box, source represents the source of the attribute information of this fused target (such as radar sensor and / or visual sensor), x and y are the radar point cloud coordinates, v x ,v y It's speed information.

[0136] It should be noted that when the target can only be detected by a visual sensor, the x, y, and v attributes of the target are fused. x ,v y The positions and velocities are estimated based on the visual detection bounding box and the calibration mapping function provided in the aforementioned embodiments (the position mapping has already been explained; the velocity calculation can be determined based on the position distance and data time difference, and will not be elaborated here). Similarly, when the target can only be detected by radar, the l, u, r, and d in the fused target attributes are the coordinates of the upper left and lower right corners of the radar detection bounding box estimated based on the radar position and the target mapping model in the aforementioned embodiments.

[0137] Therefore, even when facing unrelated radar targets and unrelated visual targets in the target detection results, sufficient attribute information can still be obtained to accurately achieve the continuous detection process of targets.

[0138] In the continuous detection phase, a two-stage data matching and association process is also required. Unlike the matching and association process in the previous embodiments, which mainly associate radar targets and visual targets in the same frame, continuous detection mainly matches and associates the fused targets of historical frames and the current frame. However, the specific principle of the two-stage association in the continuous detection phase is similar to that in the previous embodiments, which are both associations completed in two different coordinate systems. Therefore, the description of this embodiment and subsequent processes can be similarly applied to the description of the previous embodiments.

[0139] For example, reference can be made to, such as Figure 4 As shown, Figure 4 This is an exemplary continuous detection flowchart of the multi-sensor-based target detection method of this application.

[0140] At the start of the continuous detection process, the measurement target D to be associated, obtained from the current frame, needs to be input. bev The optimal estimate T of the historical frame trajectory to be associated bev The specific content of the measurement and trajectory information can be integrated with the fusion target S. F The structure is the same, including {x,y,v} x ,v y ,l,u,r,d,source}.

[0141] To ensure that the historical target detection results of historical frames are matched and associated with the current target detection results of the current frame, ignoring the temporal factor, T bev The Kalman prediction equation can be used to predict the position T′ of the historical frame trajectory in the current frame. bev For example, by inputting the historical radar position of a target object from a historical frame into the Kalman prediction equation, the predicted radar position of that target object in the current time period can be obtained. Its mathematical expression can be:

[0142] T′ bev =AT bev

[0143]

[0144] Where A is the state transition matrix in the Kalman prediction equation, and dt is the time interval between two frames.

[0145] Then, referring to the Euclidean distance model provided in the foregoing embodiments, the Euclidean distance between each trajectory prediction value (predicted radar position) and each measurement value (current radar position) can be calculated. Similarly, an association matrix between each historical target object and each current target object can be constructed, and the calculated Euclidean distance can be used as the degree of association between each historical target object and each current target object in the association matrix. The method in the foregoing embodiments can also be used to perform maximum value replacement processing on the elements in the association matrix, which will not be elaborated here.

[0146] Furthermore, by using the Hungarian method to solve the correlation matrix, the result M of the first successful match in radar coordinates can be obtained. bev (i.e., the first matched object), and the remaining unmatched objects are the first unmatched objects. It can be understood that the first unmatched objects can include the first unmatched historical objects that failed to match in historical frames and / or the first unmatched current objects that failed to match in the current frame.

[0147] Will not be in Mbev The unrelated measurement data (equivalent to the first unmatched current object) and the unrelated predicted trajectory (equivalent to the first unmatched historical object) are respectively placed into D. pv and T pv Then, similarly referring to the visual detection box matching process in the image coordinate system in the aforementioned embodiments, the second stage of matching association can be started.

[0148] Similarly, referring to the IOU distance model provided in the aforementioned embodiments, the IOU distance between the trajectory prediction value (predicted visual position, i.e., predicted visual box) of each first unmatched historical object and the measurement value (current visual position, i.e., current detection box) of each first unmatched current object can be calculated, and an association matrix can be constructed.

[0149] The correlation matrix can be solved using the Hungarian method to obtain the result M of the second matching in the image coordinate system. pv (Second matching object).

[0150] Therefore, the continuous detection results of the current frame and historical frames are completed, and the associated result (continuous detection result) M = M can be output based on the first matching object and the second matching object. bev +M pv .

[0151] Based on the above embodiments, this application embodiment describes the steps for predicting the predicted radar positions of each historical target object in the current time period. Specifically, the method of this embodiment includes the following steps:

[0152] Obtain radar sensor errors and visual sensor errors; determine the covariance matrix and measurement noise matrix based on the radar sensor errors and visual sensor errors; determine the observation matrix based on the historical target data source; construct a Kalman prediction model based on the covariance matrix, measurement noise matrix, observation matrix, and the obtained process noise matrix; input the historical radar position into the Kalman prediction model to obtain the predicted radar position output by the Kalman prediction model.

[0153] Referring to the foregoing embodiments, during continuous target detection, a time delay exists between the previous and current frame data. A common method to address this delay is to predict the trajectory of the previous frame based on a motion model, and then correlate the predicted current frame position with the actual detection position. Therefore, the quality of the motion model directly affects the accuracy of the predicted position, thus impacting the effectiveness of continuous detection. This is especially true when multiple sensors are used for collaborative detection, further increasing the difficulty of motion model estimation.

[0154] The motion modeling method in this application is consistent with other methods in the field in terms of basic ideas, mainly based on the Kalman prediction model. The state vector of the Kalman constant velocity model is usually defined as follows:

[0155] T bev ={x,y,v x ,v y}

[0156] However, unlike other methods in this field, in some application scenarios (such as road vehicle detection), due to the unique perspective of roadside data acquisition equipment and the joint detection by multiple sensors, the general definition of Kalman filter parameters is difficult to fully utilize information from multiple sensors. The mathematical expression of a common Kalman filter update equation can be as follows:

[0157] P′=APA T +Q

[0158] K = P'H T (HP'H T +R) -1

[0159] T bev =T′ bev +K(D bev -HT′ bev )

[0160] P=(I-KH)P′

[0161] Where P is the optimal estimated covariance matrix of the previous time step (historical frame), P ′ Let P represent the prior estimate covariance matrix at the current time (current frame), Q be the process noise covariance matrix, and K be the Kalman gain matrix. From the Kalman filter update equation, it can be seen that the prior estimate covariance matrix P, the observation matrix H aligning the actual measurements with the state variables, and the sensor measurement noise R all significantly affect the filter update, and these parameters are also correlated with the specific sensor. For example, radar sensors have higher measurement accuracy at long distances, while vision sensors have higher measurement accuracy at close range. Therefore, for multiple sensor information on a target object, rationally selecting the most accurate data is crucial to fully leveraging the advantages of multiple sensors.

[0162] In this application, to accurately distinguish the positioning accuracy of radar and vision at different locations, it is necessary to model the target distance and sensor positioning error. Then, Kalman adaptive parameters are constructed based on this positioning error model.

[0163] On the one hand, a radar sensor error model is established. Radar sensors detect targets by actively emitting millimeter waves, which are then reflected by objects. During reflection, the larger the target, the more difficult it is to determine its precise location based on the reflection point. Therefore, the radar sensor error is related to the actual size of the target. Considering that the target in the detection scenario is almost always moving in either direction, its reflection point is usually a point on the lower half of the target. Therefore, this application reduces the radar sensor error to half the actual length of the target. Taking a motor vehicle as an example, the radar sensor error E... R The mathematical expression can be:

[0164]

[0165] Where l and w represent the length and width of the target object (motor vehicle).

[0166] On the other hand, a visual sensor error model is established. For visual sensors, a key characteristic is that the farther the target, the smaller the pixel size, leading to a larger error in detecting distant targets. Therefore, the visual detection error at close range is much lower than that of radar sensors, while the visual detection error at distant targets is much higher. Based on the detection characteristics of visual sensors, a visual sensor error model E is constructed. V The mathematical expression can be:

[0167] E V (Y)=Yh calib -(Y+α)h calib

[0168]

[0169] The principle of the visual sensor error model is mainly to calculate the distance between two pixels in the BEV coordinate system. Here, Y represents the pixel coordinate to be calculated, and h... calib The calibration mapping matrix is ​​from the image coordinate system to the BEV coordinate system. Considering that the visual detection box may have jitter within a few pixels, the visual sensor error model of this application also needs to add the horizontal and vertical jitter range α during target detection when calculating the error. The jitter range can be a preset empirical value, which is not limited here.

[0170] In summary, with the measurement error models of the two sensors, given any location, the positioning error of each sensor can be obtained, that is, the radar sensor error and the vision sensor error can be obtained.

[0171] Furthermore, we continue to construct the Kalman adaptive parameters. Based on the Kalman update equation in the previous example, the parameters that need to be initialized are the prior estimated covariance matrix P, the observation matrix H, the measurement noise matrix R, and the process noise covariance matrix Q. The P parameter is the most important; an unreasonable P parameter will prevent the model from converging. Therefore, to allow the P parameter to switch to an appropriate parameter based on the detection situation, it can be initialized based on the target's initial position combined with the errors of the radar sensor and the visual sensor. Its mathematical expression can be:

[0172]

[0173] Where E represents the minimum measurement error between the two sensors at a specified position Y. v The speed measurement error is represented by a value of 1 when the target's data source is a radar sensor. When the target's data source is a visual sensor (equivalent to "other"), the speed measurement error is the maximum value (this maximum value is a preset value that is set towards infinity, and the larger the better). This is because the speed information from a visual sensor is usually not as accurate as that from a radar sensor, so the speed from a visual sensor can be considered unreliable.

[0174] The significance of the observation matrix H is to transform the predicted quantities into the same vector space as the observed quantities, thereby using the observed quantities to correct the predicted quantities. However, in multi-sensor scenarios, radar sensors and vision sensors not only differ in detection accuracy but also in the amount of observations. Therefore, the mathematical expression of the observation matrix H can be:

[0175]

[0176] Therefore, during the update process, different observation matrices will be selected based on the sensor source.

[0177] Similarly, the measurement noise of different sensors also varies, and the mathematical expression of the measurement noise matrix R can be:

[0178]

[0179] During the update process, the measurement noise matrix can also be adaptively adjusted according to different sensor sources.

[0180] The final process noise matrix Q represents the noise of the entire system and is independent of any single sensor. Therefore, matrix Q can be set according to the general method of Kalman filters in this field, and will not be elaborated here.

[0181] Therefore, a Kalman prediction model is constructed based on the covariance matrix (prior estimate of the covariance matrix P), the measurement noise matrix R, the observation matrix H, and the obtained process noise matrix Q. By inputting the historical radar positions into the Kalman prediction model, the predicted radar positions output by the Kalman prediction model can be obtained.

[0182] Based on the above embodiments, this application embodiment describes the steps for determining the covariance matrix and measurement noise matrix according to the radar sensor error and the visual sensor error. Specifically, the method of this embodiment includes the following steps:

[0183] Compare the radar sensor error with the visual sensor error; in response to the radar sensor error being less than the visual sensor error, determine the matrix elements in the covariance matrix and the measurement noise matrix based on the radar sensor error; in response to the radar sensor error being greater than or equal to the visual sensor error, determine the matrix elements in the covariance matrix and the measurement noise matrix based on the visual sensor error.

[0184] In conjunction with the foregoing embodiments, during the process of determining the covariance matrix and measurement noise matrix based on the radar sensor error and the visual sensor error, the corresponding sensor error can be adaptively selected to determine the matrix element E in the prior estimated covariance matrix P and the measurement noise matrix R based on the comparison results of the magnitudes of the radar sensor error and the visual sensor error.

[0185] For example, when the radar sensor error is less than the visual sensor error, according to the radar sensor error E R Determine the matrix elements E in the covariance matrix and the measurement noise matrix. When the radar sensor error is greater than or equal to the vision sensor error, determine the vision sensor error E. V Determine the matrix element E in the covariance matrix and the measurement noise matrix.

[0186] Based on the above embodiments, this application embodiment describes the steps after determining the continuous detection result according to the first matching object and the second matching object. Specifically, the method of this embodiment includes the following steps:

[0187] Retrieve unmatched historical objects from each historical target object, excluding the first and second matched objects; retrieve the number of matching failures for each unmatched historical object; delete the unmatched historical object if the number of matching failures exceeds a preset threshold; and increment the number of matching failures for each unmatched historical object if the number of matching failures is less than or equal to the preset threshold.

[0188] Referring to the foregoing embodiments, in the continuous target detection phase, unmatched objects (such as unmatched historical objects and / or unmatched current objects) can be detected. The reasons for matching failures may include environmental factors, sensor hardware malfunction, the target object leaving the detection scene, and / or the target object just entering the detection scene. Therefore, this application can count the number of matching failures for each target object (or each unmatched object), thereby revealing the reasons for the matching failures of unmatched objects.

[0189] For example, reference can be made to, such as Figure 5 As shown, Figure 5 This is an exemplary flowchart illustrating the multi-sensor-based target detection method of this application, which combines single-frame detection and continuous detection.

[0190] 1. First, input the radar detection target (such as the current radar data) and the visual detection target (such as the current visual data) of the current frame.

[0191] 2. Use the radar-visual matching method for the current frame in the aforementioned embodiments to associate radar targets and visual targets, and supplement the attribute information of the fused targets generated after association.

[0192] 3. If the current frame is the first frame in the data stream, there is no historical data in the continuous detection list. If the current frame is not the first frame in the data stream, the Kalman motion model in the aforementioned embodiment can be used to predict the motion of the targets in the continuous detection list (equivalent to the targets in the previous frame), and the predicted positions of all targets in the continuous detection list in the current frame can be obtained.

[0193] 4. Based on the prediction information of all targets in the previous frame (historical frame) and the detection information (measurement information) of the targets in the current frame, perform secondary association using the continuous detection method provided in the aforementioned embodiments.

[0194] 5. Obtain the matching results associated by the continuous detection method. Here are three examples of matching results: First: Successfully matched targets. Second: Current frame targets that did not match successfully. Third: Previous frame targets that did not match successfully (historical targets).

[0195] 6. For a successfully matched target, the position, velocity, and other attribute information of the same target in the continuous detection list need to be updated using the detection results of the target fused in the current frame, so that motion prediction can continue in the next frame of the current frame.

[0196] 7. Then, the fusion ID of the target object from the previous frame can be assigned to the fusion target of the current frame to identify that this is the same target in the previous frame and the current frame. After the ID assignment is completed, the continuous detection results can be output.

[0197] 8. For targets that fail to match in the current frame fusion, these targets may have just entered the detection scene (i.e., there is no historical data), so these targets can be added to the continuous detection list for recording.

[0198] 9. For targets from the previous frame that failed to match, these targets may have left the detection scene (i.e., there is no current data) or the current data cannot be obtained due to sensor malfunction or other reasons. Therefore, it is necessary to determine whether the number of times the target was not associated (the number of times the matching failed) exceeds the preset threshold.

[0199] 10. If the number of times the data is not associated exceeds (e.g., greater than) the preset threshold, it is considered that the fusion target has disappeared (left the detection scenario), and therefore it can be removed from the continuous detection list and no longer be continuously detected.

[0200] 11. If the number of unassociated matches does not exceed (e.g., less than or equal to) the preset threshold, it can be assumed that the target may still exist, but is not currently detected (e.g., obscured or for other reasons). Therefore, the unassociated match count (i.e., the number of failed matches) of the target is incremented by one until the number of failed matches exceeds the preset threshold.

[0201] 12. Thus, the target detection process based on the current frame and the continuous detection process based on the current frame and historical frames are completed, and the detection results can be output.

[0202] Based on the above embodiments, this embodiment can also provide a summary example of the complete process in the overall system. (See reference...) Figure 6 , Figure 6 This is an exemplary summary flowchart of the multi-sensor-based target detection method of this application.

[0203] 1. Program Startup. 2. System Initialization, Setting Relevant Thresholds. 3. Camera Intrinsic and Extrinsic Parameters / Four-Point Marking: Configure the parameters relied upon by the radar-visual matching calibration method. Subsequent radar target detection and visual target detection can be performed separately. 4. Visual Target Detection: Artificial intelligence methods such as deep learning can be used to detect the number and location of targets, assigning each target a unique visual detection ID. 5. Continuous Visual Target Detection: The purpose of this step is to ensure that the same target has a stable ID during continuous detection. 6. Extracting Detection Box Location Points: Based on the location of the detection boxes in the video, extract the center point of the detection boxes. The center point of the detection boxes can be mapped to its actual location in the world coordinate system using a calibration algorithm. 7. Radar Target Detection: The radar sensor can measure radar point cloud data in the environment, including point cloud data from real targets and point cloud data from other false targets. 8. Point Cloud Clustering: Density clustering methods can be used to obtain the number of targets and the point cloud data corresponding to each target. 9. Extracting Radar Target Regions and Determining Radar Location. The rectangular outer contour of each radar target point cloud data can be determined, and the actual position of the radar target in the world coordinate system can be calculated. 10. Time Synchronization. Synchronization processing based on data timestamps can obtain paired radar and visual data. 11. Fusion of Visual and Radar Targets. Based on the radar-visual target secondary matching algorithm in the previous example, all fused targets in the current frame are obtained. 12. Based on the continuous detection algorithm in the previous example, ID values ​​are assigned to all fused targets in the current frame according to the data of historical frames. 13. End.

[0204] In summary, the multi-sensor-based target detection method of this application performs matching and association operations in both coordinate systems during the target detection process. Therefore, even if a single sensor fails, association will still be performed in the coordinate system of the normally functioning sensor, without affecting the tracking accuracy. For the continuous detection phase, it is performed in the form of target fusion, so even if a single sensor fails, it does not affect the normal operation of the entire system. It can obtain the detection box of the radar target in the visual coordinate system through fitting, enabling the radar to be compared with the visual target at the same scale even with only sparse points, thus improving the accuracy of target detection. Because this application can perform matching and association of radar and visual targets in multiple coordinate systems, the rigor of the matching and association process is improved, thus reducing the requirements for sensor detection accuracy and calibration accuracy, expanding the application scenarios of this method. Furthermore, this application uses a sensor detection error model, which can distinguish the detection accuracy of radar and visual sensors at different distances, resulting in higher accuracy for multi-sensor joint localization. In this application, an adaptive Kalman filter is proposed. Because the motion model parameters are initialized and updated using a detection error model, it can select the more accurate parameters from the two sensors based on the sensor characteristics and the specific application scenario, thereby improving the accuracy of motion model prediction.

[0205] It should be further noted that the execution entity of the multi-sensor-based target detection method can be a multi-sensor-based target detection device. For example, the multi-sensor-based target detection method can be executed by a terminal device, a server, or other processing devices. The terminal device can be a user equipment (UE), computer, mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, the multi-sensor-based target detection method can be implemented by a processor calling computer-readable instructions stored in memory.

[0206] Figure 7 This is a block diagram illustrating a multi-sensor-based target detection device as shown in an exemplary embodiment of this application. Figure 7 As shown, this exemplary multi-sensor-based target detection device 700 includes: an acquisition module 710, a first matching module 720, a second matching module 730, and a result determination module 740. Specifically:

[0207] The acquisition module 710 is used to acquire the current visual data and the current radar data in the same scene.

[0208] The first matching module 720 is used to match the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain the first matched data and the first unmatched data. The first unmatched data includes the currently unmatched visual data and the currently unmatched radar data.

[0209] The second matching module 730 is used to match the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located, so as to obtain the second matching data.

[0210] The result determination module 740 determines the current target detection result based on the first matching data and the second matching data.

[0211] In this exemplary multi-sensor-based target detection device, by acquiring and analyzing the current visual data collected by the camera and the current radar data collected by the radar in the same scene during the current time period, the accuracy reduction of target detection caused by the limited precision of a single sensor can be compensated for. Then, the same object represented by the two types of data can be determined by the radar-visual matching method. Specifically, the current visual data and the current radar data are matched in the radar coordinate system where the radar data is located to obtain the first matched data that is successfully matched in the radar data dimension and the first unmatched data that is not matched. The first unmatched data includes the currently unmatched visual data and the currently unmatched radar data. The currently unmatched visual data and the currently unmatched radar data are matched in the image coordinate system where the visual data is located to obtain the second matched data that is successfully matched in the visual data dimension. The target detection result is determined based on the first matched data and the second matched data. Thus, this application can realize the comparison and correlation of radar data and visual data at their respective scales through a two-stage matching method, thereby improving the accuracy of target detection. Even if one of the sensors fails, the correlation will still be performed in the coordinate system corresponding to the normally functioning sensor, without affecting the change in detection accuracy.

[0212] It should be noted that the apparatus and method provided in the above embodiments belong to the same concept, and the specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the apparatus provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation.

[0213] The functions of each module can be found in the implementation examples of the target detection method based on multiple sensors, and will not be repeated here.

[0214] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of an embodiment of the electronic device of this application. The electronic device 100 includes a memory 101 and a processor 102. The processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above embodiments of the multi-sensor-based target detection method. In a specific implementation scenario, the electronic device 100 may include, but is not limited to, a microcomputer or a server. In addition, the electronic device 100 may also include mobile devices such as laptops and tablets, which are not limited here.

[0215] Specifically, processor 102 controls itself and memory 101 to implement the steps in any of the above-described embodiments of the multi-sensor-based target detection method. Processor 102 can also be referred to as a CPU (Central Processing Unit). Processor 102 may be an integrated circuit chip with signal processing capabilities. Processor 102 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 102 can be implemented using integrated circuit chips.

[0216] In this exemplary electronic device, by acquiring and analyzing the current visual data collected by the camera and the current radar data collected by the radar in the same scene during the current time period, the reduced target detection accuracy caused by the limited precision of a single sensor can be compensated for. Then, the same object represented by the two types of data can be determined through a radar-visual matching method. Specifically, the current visual data and current radar data are matched in the radar coordinate system where the radar data resides, resulting in first matched data (successfully matched) and first unmatched data (failing to match) in the radar data dimension. The first unmatched data includes both currently unmatched visual data and currently unmatched radar data. The currently unmatched visual data and currently unmatched radar data are then matched in the image coordinate system where the visual data resides, resulting in second matched data (successfully matched) in the visual data dimension. The target detection result is determined based on the first and second matched data. Therefore, this application can achieve comparative correlation between radar data and visual data at their respective scales through a two-stage matching method, improving the accuracy of target detection. Even if a single sensor malfunctions, correlation will still occur within the coordinate system of the normally functioning sensor, without affecting the change in detection accuracy.

[0217] Please see Figure 9 , Figure 9 This is a schematic diagram of a computer-readable storage medium according to an embodiment of the present application. The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor. The program instructions 111 are used to implement the steps in any of the above embodiments of the multi-sensor-based target detection method.

[0218] In this exemplary storage medium, by running program instructions within the storage medium, the current visual data acquired by the camera and the current radar data acquired by the radar in the same scene during the current time period are obtained and analyzed. This can compensate for the reduced target detection accuracy caused by the limited precision of a single sensor. Then, the same object represented by the two types of data can be determined through a radar-visual matching method. Specifically, the current visual data and the current radar data are matched in the radar coordinate system where the radar data is located, resulting in first matched data and first unmatched data that are successfully matched in the radar data dimension. The first unmatched data includes currently unmatched visual data and currently unmatched radar data. The currently unmatched visual data and the currently unmatched radar data are matched in the image coordinate system where the visual data is located, resulting in second matched data that are successfully matched in the visual data dimension. The target detection result is determined based on the first and second matched data. Thus, this application can achieve comparative correlation between radar data and visual data at their respective scales through a two-stage matching method, improving the accuracy of target detection. Even if a single sensor malfunctions, correlation will still be performed in the coordinate system corresponding to the normally functioning sensor, without affecting the change in detection accuracy.

[0219] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0220] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0221] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0222] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A target detection method based on multiple sensors, characterized in that, The method includes: Acquire current visual data and current radar data in the same scene; The current visual data and the current radar data are matched in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data. The first unmatched data includes the current unmatched visual data and the current unmatched radar data. The currently unmatched visual data and the currently unmatched radar data are matched in the image coordinate system where the visual data is located to obtain the second matching data; The current target detection result is determined based on the first matching data and the second matching data.

2. The method according to claim 1, characterized in that, The step of matching the currently unmatched visual data and the currently unmatched radar data in the image coordinate system where the visual data is located to obtain the second matched data includes: Obtain the target mapping model; Generate the current radar detection box corresponding to the current unmatched radar data in the image coordinate system based on the target mapping model; The current radar detection box is matched with the current unmatched visual detection box of the current unmatched visual data to obtain the second matching data.

3. The method according to claim 2, characterized in that, The acquisition of the target mapping model includes: Obtain the current matching visual data and the current matching radar data from the first matching data. The current matching visual data includes the size information of the current matching visual detection box, and the current matching radar data includes radar distance information. The target mapping model is obtained by fitting the size information and the radar distance information.

4. The method according to claim 2, characterized in that, The step of matching the current radar detection box with the current unmatched visual data to obtain the second matching data includes: The degree of association between each radar detection box and each currently unmatched visual detection box is determined based on the cross-union ratio between each radar detection box and each currently unmatched visual detection box. The second matching data is determined from each currently unmatched visual data and each currently unmatched radar data based on the degree of correlation.

5. The method according to claim 1, characterized in that, The step of matching the current visual data and the current radar data in the radar coordinate system where the radar data is located to obtain first matched data and first unmatched data includes: The current visual position of the current visual data is transformed into the radar coordinate system to obtain the visual transformation position; The degree of correlation between each current visual data and each current radar data is determined based on the distance between the visual conversion position and the current radar position of the current radar data. The first matching data and the first unmatched data are determined from each current visual data and each current radar data based on the degree of correlation.

6. The method according to claim 5, characterized in that, The step of determining the first matching data and the first unmatched data from each current visual data and each current radar data according to the degree of correlation includes: Based on the degree of correlation between each current visual data and each current radar data, construct a correlation matrix between each current visual data and each current radar data. The first matching data is determined based on the correlation matrix and a preset minimum weight matching algorithm; Each current visual data and each current radar data, excluding the first matching data, are identified as the first unmatched data.

7. The method according to claim 1, characterized in that, The current target detection result includes the current target object. After determining the current target detection result based on the first matching data and the second matching data, the method further includes: Acquire historical target detection results detected in historical time periods, wherein the historical time periods are earlier than the current time period corresponding to the current target detection result, and the historical target detection results include historical target objects; Predict the location of each historical target object corresponding to each predicted radar in the current time period; Based on the predicted radar positions and the current radar positions, each historical target object is matched with each current target object to obtain a first matched object and a first unmatched object. The first unmatched object includes a first unmatched historical object and a first unmatched current object. Predict the predicted visual locations of the first unmatched historical object in the current time period; Based on each predicted visual position and each current visual position, each first unmatched historical object and each first unmatched current object are matched to obtain the second matched object; The continuous detection result is determined based on the first matching object and the second matching object.

8. The method according to claim 7, characterized in that, The predicted radar positions for each historical target object in the current time period include: Acquire radar sensor error and vision sensor error; The covariance matrix and measurement noise matrix are determined based on the radar sensor error and the visual sensor error; Determine the observation matrix based on the data source of the historical target object; A Kalman prediction model is constructed based on the covariance matrix, the measurement noise matrix, the observation matrix, and the obtained process noise matrix. The historical radar locations are input into the Kalman prediction model to obtain the predicted radar locations output by the Kalman prediction model.

9. The method according to claim 8, characterized in that, The step of determining the covariance matrix and measurement noise matrix based on the radar sensor error and the vision sensor error includes: Compare the radar sensor error with the visual sensor error; In response to the radar sensor error being less than the visual sensor error, matrix elements in the covariance matrix and the measurement noise matrix are determined based on the radar sensor error; In response to the radar sensor error being greater than or equal to the visual sensor error, matrix elements in the covariance matrix and the measurement noise matrix are determined based on the visual sensor error.

10. The method according to claim 7, characterized in that, After determining the continuous detection result based on the first matching object and the second matching object, the method further includes: Obtain the unmatched historical objects from each historical target object, excluding the first matching object and the second matching object; Get the number of matching failures for the unmatched historical object; In response to the number of failed matches exceeding a preset threshold, the unmatched historical objects are deleted. In response to the number of matching failures being less than or equal to the preset number threshold, the number of matching failures for the unmatched historical objects is accumulated.

11. An electronic device, characterized in that, The method includes a memory and a processor, the processor being configured to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, they implement the method described in any one of claims 1 to 10.

Images