Method and device for evaluating fusion perception algorithm, storage medium and vehicle

Abstract

This disclosure relates to a method, apparatus, storage medium, and vehicle for evaluating fusion perception algorithms. The method includes: acquiring point cloud data obtained from a millimeter-wave radar sensor perceiving a target scene, and a perceived image obtained from a camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view; performing target recognition on the point cloud data and the perceived image according to the fusion perception algorithm to be evaluated, obtaining the identified perceived target; associating real targets in the target scene with the perceived target, obtaining an association result; and determining the evaluation result of the fusion perception algorithm to be evaluated based on the association result. This method can determine the merits of the fusion perception algorithm to be evaluated.

Description

Technical Field

[0001] This disclosure relates to the field of target detection and recognition technology, and in particular to a method, apparatus, storage medium, and vehicle for evaluating fusion perception algorithms. Background Technology

[0002] Autonomous driving systems perceive the vehicle's surroundings through a perception system and make driving decisions based on this information to control the vehicle's autonomous driving. During this process, the perception system performs entity / target recognition, such as obstacle recognition, on the sensor-collected data. Based on the entity / target recognition results, the autonomous driving system can make driving decisions adapted to the current environment.

[0003] In related technologies, the quality of target recognition algorithms (including algorithms for target recognition from a single perception data source and algorithms for target recognition from multiple perception data sources, i.e., fused data sources) affects the accuracy of target recognition results, which in turn has a significant impact on whether autonomous driving systems make safe driving decisions. Therefore, to ensure the safe operation of autonomous vehicles, it is necessary to evaluate the quality of target recognition algorithms. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a method, apparatus, storage medium, and vehicle for evaluating fusion perception algorithms.

[0005] According to a first aspect of the present disclosure, a method for evaluating a fusion perception algorithm is provided, the method comprising:

[0006] The system acquires point cloud data obtained by a millimeter-wave radar sensor perceiving a target scene, and a perceived image obtained by a camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view.

[0007] The point cloud data and the perceived image are used to perform target recognition based on the fusion perception algorithm to be evaluated, and the recognized perceived target is obtained.

[0008] The real targets in the target scene are associated with the perceived targets to obtain the association results;

[0009] The evaluation result of the fusion perception algorithm to be evaluated is determined based on the correlation results.

[0010] Optionally, the number of perceived targets is multiple, and the number of real targets is multiple. The step of associating the real targets in the target scene with the perceived targets to obtain an association result includes:

[0011] Based on the data source type corresponding to each of the aforementioned sensing targets, all the sensing targets are classified to obtain a set of sensing targets corresponding to different data source types;

[0012] For each set of perception targets, according to the association processing strategy corresponding to the data source type of the set of perception targets, the real target is associated with each of the perception targets in the set of perception targets to obtain the association processing sub-result corresponding to the set of perception targets;

[0013] The association results include the association processing sub-results corresponding to each set of perception targets.

[0014] Optionally, if the data source type of the perceived target set is an image data source type, the real target and each perceived target in the perceived target set are associated with each other according to the association processing strategy corresponding to the image data source type, to obtain the association processing sub-result corresponding to the perceived target set, including:

[0015] For each of the perceived targets in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each of the real targets is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target.

[0016] For each of the sensing targets in the set of sensing targets, the area intersection-union ratio (IUU) between the sensing target and each of the real targets is calculated to obtain an area IUU matrix. The rows of the area IUU matrix correspond to one of the sensing target and the real target, and the columns of the area IUU matrix correspond to the other of the sensing target and the real target.

[0017] Based on the size of each element in the angle intersection-union matrix and the size of each element in the area intersection-union matrix, the associated perceived targets and the associated real targets are determined. The associated processing sub-result includes the associated perceived targets and the associated real targets.

[0018] Optionally, when the rows of the angle intersection-union matrix correspond to the perceived target, and the rows of the area intersection-union matrix correspond to the perceived target, determining the associated perceived target and the associated real target based on the size of each element in the angle intersection-union matrix and the size of each element in the area intersection-union matrix includes:

[0019] For each of the perceived targets, a first target element greater than a first preset threshold is determined from the row corresponding to the perceived target in the intersection-union matrix; and,

[0020] The real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix are determined as candidate real targets;

[0021] Determine the second target element corresponding to each candidate real target from the rows corresponding to the perceived target in the area intersection-union matrix;

[0022] Determine the maximum value among all the second target elements, and if the maximum value is greater than a second preset threshold, determine the candidate real target corresponding to the maximum value as the associated real target that is mutually associated with the perceived target, and determine the perceived target as the associated perceived target.

[0023] Optionally, the method further includes:

[0024] If the maximum value is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets.

[0025] If the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0026] Optionally, if the data source type of the perceived target set is an image data source type, the real target and each perceived target in the perceived target set are associated with each other according to the association processing strategy corresponding to the image data source type, to obtain the association processing sub-result corresponding to the perceived target set, including:

[0027] For each of the perceived targets in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each of the real targets is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target.

[0028] When a row in the intersection-union-matrix (IUCM) corresponds to a sensing target, for each sensing target, a first target element greater than a first preset threshold is determined from the row corresponding to the sensing target in the IUCM; and,

[0029] The real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix are determined as candidate real targets;

[0030] Calculate the area intersection-union ratio (IUU) of the perceived target and each of the candidate real targets, and determine the target with the largest IUU value;

[0031] If the intersection-union ratio of the target area is greater than the second preset threshold, the candidate real target corresponding to the intersection-union ratio of the target area is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0032] Optionally, the method further includes:

[0033] If the intersection-union ratio of the target area is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets.

[0034] If the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0035] Optionally, the intersection-union ratio (IU) of the angle between the perceived target and the real target is calculated as follows:

[0036] Calculate the first included angle range of the perception target bounding box corresponding to the perception target in the target coordinate system. The perception target bounding box is determined based on the position of the perception target in the perception image.

[0037] Calculate the second included angle range of the real target bounding box corresponding to the real target in the target coordinate system, where both the first included angle range and the second included angle range are included angle ranges relative to the X-axis;

[0038] The intersection-union ratio of the angles between the perceived target and the real target is calculated based on the first angle range and the second angle range.

[0039] Optionally, if the data source type of the sensing target set is a fusion data source type, the real target and each sensing target in the sensing target set are associated with each other according to the association processing strategy corresponding to the fusion data source type, to obtain the association processing sub-result corresponding to the sensing target set, including:

[0040] For each of the perceived targets in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each of the real targets is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target.

[0041] For each of the sensing targets in the set of sensing targets, the area intersection-union ratio (IUU) between the sensing target and each of the real targets is calculated to obtain an area IUU matrix. The rows of the area IUU matrix correspond to one of the sensing target and the real target, and the columns of the area IUU matrix correspond to the other of the sensing target and the real target.

[0042] The target matrix is ​​obtained by weighted calculation based on the included angle intersection-to-union matrix and the area intersection-to-union matrix;

[0043] Based on the size of each element in the target matrix, the associated perceived targets and the associated real targets are determined. The associated processing sub-result includes the associated perceived targets and the associated real targets.

[0044] Optionally, when the rows of the intersection-union matrix of the included angle correspond to the perceived target, and the rows of the intersection-union matrix of the area correspond to the perceived target, determining the associated perceived target and the associated real target based on the size of each element in the target matrix includes:

[0045] For each of the sensing targets in the set of sensing targets, determine the target element with the largest value from the row corresponding to the sensing target in the target matrix;

[0046] When the target element is greater than the fourth preset threshold, the real target corresponding to the column where the target element is located is determined as the associated real target that is related to the perceived target, and the perceived target is determined as the associated perceived target;

[0047] If the target element is less than or equal to the fourth preset threshold, a first target element greater than the fifth preset threshold is determined from the row corresponding to the perceived target in the angle intersection-union matrix.

[0048] The real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix are determined as the first candidate real targets;

[0049] Calculate the confidence level between the perceived target and each of the first candidate real targets, and determine the second candidate real target corresponding to the confidence level that is greater than a sixth preset threshold;

[0050] The second candidate real target with the smallest longitudinal error rate with the perceived target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0051] Optionally, the method further includes:

[0052] The point cloud data is used to identify targets according to the fusion perception algorithm to be evaluated, and a first set of perception targets is obtained. Each first perception target in the first set of perception targets corresponds to a point cloud data source type.

[0053] The target recognition is performed on the perceived image according to the fusion perception algorithm to be evaluated, and a second set of perceived targets is obtained. The image data source type of each second perceived target in the second set of perceived targets is a different type of image data source.

[0054] For each of the aforementioned sensing targets, a first sensing target with the highest similarity to the sensing target is determined from the first set of sensing targets, and a second sensing target with the highest similarity to the sensing target is determined from the second set of sensing targets;

[0055] The data source type of the sensing target is determined based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target.

[0056] Optionally, determining the data source type of the sensing target based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target, includes:

[0057] If the first similarity is greater than the similarity threshold and the second similarity is greater than the similarity threshold, the data source type of the perceived target is determined to be a fusion data source type.

[0058] If the first similarity is less than or equal to the similarity threshold and the second similarity is greater than the similarity threshold, the data source type of the perceived target is determined to be the image data source type.

[0059] Optionally, determining the evaluation result of the fusion perception algorithm to be evaluated based on the association result includes:

[0060] The accuracy of the fusion sensing algorithm to be evaluated is determined based on the number of associated sensing targets and the total number of sensing targets.

[0061] The recall rate of the fusion perception algorithm to be evaluated is determined based on the number of associated real targets and the total number of real targets.

[0062] The evaluation results include the precision rate and the recall rate.

[0063] According to a second aspect of the present disclosure, an apparatus for evaluating a fusion perception algorithm is provided, the apparatus comprising:

[0064] The acquisition module is configured to acquire point cloud data obtained by the millimeter-wave radar sensor perceiving the target scene, and the perceived image obtained by the camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view.

[0065] The first recognition module is configured to perform target recognition on the point cloud data and the perception image according to the fusion perception algorithm to be evaluated, and obtain the recognized perception target;

[0066] The association module is configured to associate the real targets in the target scene with the perceived targets to obtain an association result;

[0067] The execution module is configured to determine the evaluation result of the fusion perception algorithm to be evaluated based on the association result.

[0068] Optionally, the number of perceived targets is multiple, the number of real targets is multiple, and the association module includes:

[0069] The classification submodule is configured to classify all the sensing targets based on the data source type corresponding to each sensing target, and obtain a set of sensing targets corresponding to different data source types;

[0070] The association submodule is configured to, for each set of perception targets, perform association processing on the real target and each of the perception targets in the set of perception targets according to the association processing strategy corresponding to the data source type of the set of perception targets, to obtain the association processing sub-result corresponding to the set of perception targets; wherein, the association result includes the association processing sub-result corresponding to each set of perception targets.

[0071] Optionally, if the data source type of the perceived target set is an image data source type, the association submodule includes:

[0072] The first calculation submodule is configured to calculate the angle intersection ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an angle intersection ratio matrix, wherein the rows of the angle intersection ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection ratio matrix correspond to the other of the perceived target and the real target.

[0073] The second calculation submodule is configured to calculate the area intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an area intersection-union ratio matrix, wherein the rows of the area intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the area intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0074] The first determining submodule is configured to determine the associated perceived target and the associated real target based on the size of each element in the angle intersection-union matrix and the size of each element in the area intersection-union matrix. The association processing sub-result includes the associated perceived target and the associated real target.

[0075] Optionally, when a row of the included angle intersection-union matrix corresponds to the sensing target, and a row of the area intersection-union matrix corresponds to the sensing target, the first determining submodule is configured as follows:

[0076] For each perceived target, a first target element greater than a first preset threshold is determined from the row corresponding to the perceived target in the angle intersection-union matrix; and the real targets corresponding to the columns where each of the first target elements in the angle intersection-union matrix is ​​located are determined as candidate real targets; a second target element corresponding to each candidate real target is determined from the row corresponding to the perceived target in the area intersection-union matrix; the maximum value among all the second target elements is determined, and if the maximum value is greater than a second preset threshold, the candidate real target corresponding to the maximum value is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0077] Optionally, the first determining submodule is further configured to:

[0078] If the maximum value is less than or equal to the second preset threshold, a first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets; if the longitudinal error rate between the first candidate real target and the perceived target is less than the third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0079] Optionally, if the data source type of the perceived target set is an image data source type, the association submodule includes:

[0080] The third calculation submodule is configured to calculate the angle intersection ratio between the perceived target and each real target for each perceived target in the set of perceived targets, and obtain an angle intersection ratio matrix, wherein the rows of the angle intersection ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection ratio matrix correspond to the other of the perceived target and the real target.

[0081] The second determining submodule is configured to, when the rows of the angle intersection-union matrix correspond to the perceived targets, determine, for each perceived target, a first target element greater than a first preset threshold from the rows corresponding to the perceived targets in the angle intersection-union matrix; and determine the real targets corresponding to the columns where each of the first target elements in the angle intersection-union matrix is ​​located as candidate real targets.

[0082] The fourth calculation submodule is configured to calculate the area intersection-union ratio of the perceived target and each of the candidate real targets, and determine the target area intersection-union ratio with the largest value;

[0083] The third determining submodule is configured to, when the intersection-union ratio of the target area is greater than a second preset threshold, determine the candidate real target corresponding to the intersection-union ratio of the target area as the associated real target that is mutually associated with the perceived target, and determine the perceived target as the associated perceived target.

[0084] Optionally, the third determining submodule is further configured to:

[0085] If the intersection-union ratio of the target area is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets; if the longitudinal error rate between the first candidate real target and the perceived target is less than the third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0086] Optionally, the intersection-union ratio (IU) of the angle between the perceived target and the real target is calculated as follows:

[0087] Calculate the first included angle range of the perception target bounding box corresponding to the perception target in the target coordinate system. The perception target bounding box is determined based on the position of the perception target in the perception image.

[0088] Calculate the second included angle range of the real target bounding box corresponding to the real target in the target coordinate system, where both the first included angle range and the second included angle range are included angle ranges relative to the X-axis;

[0089] The intersection-union ratio of the angles between the perceived target and the real target is calculated based on the first angle range and the second angle range.

[0090] Optionally, if the data source type of the perceived target set is a fused data source type, the association submodule includes:

[0091] The fifth calculation submodule is configured to calculate the angle intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an angle intersection-union ratio matrix, wherein the rows of the angle intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0092] The sixth calculation submodule is configured to calculate the area intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an area intersection-union ratio matrix, wherein the rows of the area intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the area intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0093] The weighting submodule is configured to calculate the target matrix by weighting the included angle intersection-to-union matrix and the area intersection-to-union matrix.

[0094] The fourth determination submodule is configured to determine, based on the size of each element in the target matrix, the associated perceived targets and the associated real targets, and the associated processing sub-results include the associated perceived targets and the associated real targets.

[0095] Optionally, when a row of the included angle intersection-union matrix corresponds to the sensing target, and a row of the area intersection-union matrix corresponds to the sensing target, the fourth determining submodule is configured as follows:

[0096] For each of the perceived targets in the set of perceived targets, the target element with the largest value is determined from the row corresponding to the perceived target in the target matrix; if the target element is greater than a fourth preset threshold, the real target corresponding to the column where the target element is located is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target; if the target element is less than or equal to the fourth preset threshold, a first target element with a value greater than a fifth preset threshold is determined from the row corresponding to the perceived target in the intersection-union matrix; the real targets corresponding to the columns where each first target element is located in the intersection-union matrix are determined as first candidate real targets; the confidence level between the perceived target and each first candidate real target is calculated, and a second candidate real target corresponding to the confidence level greater than a sixth preset threshold is determined; the second candidate real target with the smallest longitudinal error rate with the perceived target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0097] Optionally, the device further includes:

[0098] The second identification module is configured to perform target identification on the point cloud data according to the fusion perception algorithm to be evaluated, and obtain a first set of perception targets, wherein each first perception target in the first set of perception targets corresponds to a point cloud data source type;

[0099] The third recognition module is configured to perform target recognition on the perceived image according to the fusion perception algorithm to be evaluated, and obtain a second set of perceived targets, wherein each second perceived target in the second set of perceived targets corresponds to an image data source type;

[0100] The first determining module is configured to, for each of the perceived targets, determine a first perceived target with the highest similarity to the perceived target from the first set of perceived targets, and determine a second perceived target with the highest similarity to the perceived target from the second set of perceived targets;

[0101] The second determining module is configured to determine the data source type of the sensing target based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target.

[0102] Optionally, the second determining module includes:

[0103] The fifth determining submodule is configured to determine the data source type of the perceived target as a fusion data source type when the first similarity is greater than the similarity threshold and the second similarity is greater than the similarity threshold.

[0104] The sixth determining submodule is configured to determine the data source type of the perceived target as the image data source type when the first similarity is less than or equal to the similarity threshold and the second similarity is greater than the similarity threshold.

[0105] Optionally, the execution module is configured as follows:

[0106] The precision of the fusion perception algorithm to be evaluated is determined based on the number of associated perceived targets and the total number of perceived targets; the recall of the fusion perception algorithm to be evaluated is determined based on the number of associated real targets and the total number of real targets; wherein the evaluation result includes the precision and the recall.

[0107] According to a third aspect of the present disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the steps of the method for evaluating a fusion perception algorithm provided in the first aspect of the present disclosure.

[0108] According to a fourth aspect of the present disclosure, a vehicle is provided, comprising:

[0109] A memory on which computer programs are stored;

[0110] A processor for executing the computer program in the memory to implement the steps of the method for evaluating a fusion sensing algorithm provided in the first aspect of this disclosure.

[0111] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0112] Point cloud data obtained from a millimeter-wave radar sensor sensing a target scene, and perceived images obtained from a camera sensing the target scene, are acquired. Since the millimeter-wave radar sensor and the camera share the same sensing field of view, the same sensing objects exist in both the point cloud data and the perceived images. The point cloud data and perceived images are input into a fusion sensing algorithm to be evaluated for target recognition, resulting in identified sensing targets. Real targets in the target scene are then correlated with the sensing targets to obtain correlation results. The evaluation result of the fusion sensing algorithm is determined based on the correlation results between the real targets and the sensing targets. Using this method, the evaluation result of the fusion sensing algorithm can be calculated based on the correlation results between real targets and sensing targets, thereby determining the merits of the fusion sensing algorithm. The merits of the fusion sensing algorithm can be characterized by metrics such as precision / accuracy and recall.

[0113] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0114] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0115] Figure 1 This is a flowchart illustrating a method for evaluating a fusion sensing algorithm according to an exemplary embodiment.

[0116] Figure 2 This is a block diagram illustrating an apparatus for evaluating a fusion sensing algorithm according to an exemplary embodiment.

[0117] Figure 3 This is a functional block diagram of a vehicle illustrating an exemplary embodiment. Detailed Implementation

[0118] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0119] It should be noted that all actions involving the acquisition of signals, information, or data in this application are carried out in compliance with the relevant data protection laws and policies of the country where the application is located, and with the authorization granted by the owner of the relevant device.

[0120] Figure 1 This is a flowchart illustrating a method for evaluating a fusion sensing algorithm according to an exemplary embodiment, such as... Figure 1 As shown, this method for evaluating fusion perception algorithms is used in terminal devices. For example, it can be applied to in-vehicle terminal devices, computers, laptops, and other electronic devices. The method for evaluating fusion perception algorithms may include the following steps.

[0121] S11. Acquire point cloud data obtained by the millimeter-wave radar sensor to perceive the target scene, and the perceived image obtained by the camera to perceive the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view.

[0122] It should be explained that a point cloud represents a dataset containing multiple points. Each point can include information such as geometric coordinates (X, Y), timestamp, intensity value, velocity value, and RCS value (radar cross section). The intensity value refers to the strength of the signal received after a signal emitted by a sensor is reflected back from an object. When these points are combined, they form a point cloud.

[0123] Millimeter-wave radar (MMR) is a crucial sensor for autonomous vehicles due to its low cost, long detection range, ability to acquire speed information, and strong anti-interference capabilities. However, the point clouds acquired by MMR are too sparse and have low resolution, and MMR also cannot determine the height information of obstacles. Therefore, designing algorithms for target recognition based on point cloud data acquired by MMR sensors is difficult, and evaluating such algorithms is also challenging. In related technologies, MMR sensors are often combined with other sensors to compensate for the limitations of the data acquired by MMR sensors. For example, MMR sensors are combined with vision sensors such as cameras.

[0124] When millimeter-wave radar sensors are used in conjunction with cameras, both sensors simultaneously perceive the target scene to obtain point cloud data and perceived images. The millimeter-wave radar sensors and cameras share the same field of view, which corresponds to the range of the target scene that both the millimeter-wave radar sensors and cameras can perceive and detect.

[0125] S12. Based on the fusion perception algorithm to be evaluated, target recognition is performed on the point cloud data and the perception image to obtain the recognized perception target.

[0126] In one implementation, point cloud data and perceived images are combined using a fusion perception algorithm to be evaluated for target recognition, resulting in identified perceived targets. Specifically, target size, target location, target type / ID name, etc., can be identified. It should be noted that in this embodiment, the identified perceived targets and real targets in the target scene refer to entities such as cars, pedestrians, birds, trees, billboards, rocks, puddles, etc.

[0127] Among them, the fusion perception algorithm is a target recognition algorithm, also known as a target detection algorithm, used to fuse point cloud data and perceived images to detect and identify targets, as well as their locations, such as outlining the specific location of the target. Specifically, the fusion perception algorithm to be evaluated in this disclosure refers to an algorithm used to fuse point cloud data acquired by a millimeter-wave radar sensor and perceived images acquired by a camera for target recognition and detection.

[0128] S13. Associate the real target in the target scene with the perceived target to obtain the association result.

[0129] The real targets in the target scene can be determined by annotating the point cloud data obtained from the millimeter-wave radar sensor's perception of the target scene in step S11 and the perceived images obtained from the camera's simultaneous perception of the target scene. Alternatively, the real targets in the target scene can be determined by annotating the point cloud data obtained from the lidar sensor's perception of the target scene. Or, the real targets can be obtained by manually measuring the target scene. This disclosure does not specifically limit this. A preferred embodiment uses a pre-annotated evaluation dataset. This evaluation dataset includes the point cloud data obtained from the millimeter-wave radar sensor's perception of the target scene in step S11, the perceived images obtained from the camera's simultaneous perception of the target scene, and the annotation information of the real targets in the target scene.

[0130] S14. Determine the evaluation result of the fusion perception algorithm to be evaluated based on the association result.

[0131] In some implementations, real targets in the target scene can be associated with perceived targets to obtain mutually associated perceived targets and real targets. Mutual association can be understood as mutual matching. The quality of the fusion perception algorithm can be determined based on the number of mutually associated perceived targets and real targets, and the magnitude of the error between them. For example, the quality of the fusion perception algorithm can be calculated and determined based on its accuracy, recall, fitness, etc.

[0132] Using the above method, based on the correlation results between the real target and the perceived target, the evaluation result of the fusion perception algorithm to be evaluated can be calculated, thereby determining the merits of the fusion perception algorithm. The merits of the fusion perception algorithm to be evaluated can be characterized by indicators such as precision / accuracy and recall.

[0133] Optionally, the number of perceived targets is multiple, and the number of real targets is multiple. The step of associating the real targets in the target scene with the perceived targets to obtain an association result includes:

[0134] Based on the data source type corresponding to each of the sensing targets, all the sensing targets are classified to obtain a set of sensing targets corresponding to different data source types; for each set of sensing targets, according to the association processing strategy corresponding to the data source type of the sensing target set, the real target is associated with each of the sensing targets in the set to obtain an association processing sub-result corresponding to the set of sensing targets; wherein, the association result includes the association processing sub-result corresponding to each set of sensing targets.

[0135] It should be noted that because millimeter-wave radar sensors and cameras share the same field of view, point cloud data and perceived images can contain perception information for the same object. Similarly, point cloud data can contain perception information for locations that are blind spots for the camera, and perceived images can contain perception information for locations that are blind spots for the millimeter-wave radar sensor. Therefore, when the fusion perception algorithm under evaluation performs target recognition on point cloud data and perceived images, the perceived target may be determined based on the point cloud data, meaning the target is in the target scene and can be perceived by the millimeter-wave radar sensor but not by the camera. Alternatively, the perceived target may be determined based on the perceived image, meaning the target is in the target scene and can be perceived by the camera but not by the millimeter-wave radar sensor. Another possibility is that the perceived target is determined based on both point cloud data and perceived images, meaning the target is in the target scene and can be perceived by both the camera and the millimeter-wave radar sensor. Thus, the perceived targets obtained by the fusion perception algorithm under evaluation for target recognition and detection on point cloud data and perceived images correspond to different data source types. The data source type can be an image data source type that represents the perception target based on the perception image, a point cloud data source type that represents the perception target based on the point cloud data, or a fusion data source type that represents the perception target based on both the perception image and the point cloud data.

[0136] In some implementations, the fusion sensing algorithm to be evaluated performs target recognition and detection on point cloud data and perceived images. While obtaining the perceived targets, it can also output the data source type corresponding to each perceived target. Based on the data source type corresponding to each perceived target, all perceived targets can be classified to obtain a set of perceived targets corresponding to different data source types. Since cameras and millimeter-wave radar sensors have different characteristics in perceiving targets—for example, cameras can better perceive the lateral information of targets compared to millimeter-wave radar sensors, while millimeter-wave radar sensors can better perceive the longitudinal information of targets compared to cameras—different association processing strategies can be adopted for the association processing of the sets of perceived targets corresponding to different data source types in this embodiment.

[0137] It should be noted that in the embodiments disclosed herein, "lateral" refers to the lateral direction of the lane, and "longitudinal" refers to the longitudinal direction of the lane, that is, the direction in which the lane extends.

[0138] If the fusion perception algorithm to be evaluated performs target recognition and detection on point cloud data and perceived images, but does not output the data source type corresponding to each perceived target, and it is also impossible to know the data source type corresponding to each perceived target, then the following method can be used to determine the data source type corresponding to each perceived target.

[0139] First, target identification is performed on the point cloud data according to the fusion perception algorithm to be evaluated, and a first perception target set is obtained. Each first perception target in the first perception target set corresponds to a point cloud data source type.

[0140] Furthermore, target recognition is performed on the perceived image based on the fusion perception algorithm to be evaluated, resulting in a second set of perceived targets, and the image data source type corresponding to each second perceived target in the second set of perceived targets.

[0141] Next, for each perceived target, the first perceived target with the highest similarity to that target is determined from the first set of perceived targets, and the second perceived target with the highest similarity to that target is determined from the second set of perceived targets. The similarity can be calculated by assessing the similarity in position, velocity, size, etc., between the perceived target and each of the first / second perceived targets. For example, the perceived target and each of the first / second perceived targets can be projected onto the same coordinate system space, and then similarity calculation methods such as Mahalanobis distance and cosine similarity can be used to obtain the similarity between the perceived target and each of the first / second perceived targets.

[0142] Then, based on the magnitude of the first similarity between the perceived target and the first perceived target, and the magnitude of the second similarity between the perceived target and the second perceived target, the data source type of the perceived target is determined.

[0143] For example, if the first similarity is greater than the similarity threshold and the second similarity is greater than the similarity threshold, the data source type of the perceived target is determined to be a fusion data source type, that is, the fusion perception algorithm to be evaluated identifies the perceived target based on the information in the point cloud data and the perceived image.

[0144] For example, if the first similarity is less than or equal to a similarity threshold and the second similarity is greater than a similarity threshold, the data source type of the perceived target is determined to be an image data source type. That is, the fusion perception algorithm to be evaluated identifies the perceived target based on information in the perceived image.

[0145] For example, if the first similarity is greater than a similarity threshold and the second similarity is less than or equal to the similarity threshold, the data source type of the perceived target is determined to be a point cloud data source type. That is, the fusion sensing algorithm to be evaluated identifies the perceived target based on information in the point cloud data.

[0146] Optionally, if the data source type of the perceived target set is an image data source type, the real target and each perceived target in the perceived target set are associated with each other according to the association processing strategy corresponding to the image data source type, to obtain the association processing sub-result corresponding to the perceived target set, including the following steps:

[0147] For each perceived target in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each real target is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns correspond to the other. For each perceived target in the set of perceived targets, the intersection-union ratio (IUU) of the area between the perceived target and each real target is calculated to obtain an area IUU matrix. The rows of the area IUU matrix correspond to one of the perceived target and the real target, and the columns correspond to the other. Based on the size of each element in the IUU matrix and the size of each element in the area IUU matrix, associated perceived targets and associated real targets are determined. The associated processing sub-result includes the associated perceived targets and the associated real targets.

[0148] For example, suppose the set of perceived targets, whose data source type is image data source, contains two perceived targets A and B. All real targets are C, D, and E. Then, for perceived target A, calculate the intersection-union ratio (IU) X(A, C) between perceived target A and real target C. Calculate the IU X(A, D) between perceived target A and real target D. Calculate the IU X(A, E) between perceived target A and real target E. And, for perceived target B, calculate the IU X(B, C) between perceived target B and real target C. Calculate the IU X(B, D) between perceived target B and real target D. Calculate the IU X(B, E) between perceived target B and real target E. The resulting IU matrix is: or .

[0149] Similarly, for perceived target A, calculate the area intersection-union ratio Y(A, C) between perceived target A and real target C. Calculate the area intersection-union ratio Y(A, D) between perceived target A and real target D. Calculate the area intersection-union ratio Y(A, E) between perceived target A and real target E. Furthermore, for perceived target B, calculate the area intersection-union ratio Y(B, C) between perceived target B and real target C. Calculate the area intersection-union ratio Y(B, D) between perceived target B and real target D. Calculate the area intersection-union ratio Y(B, E) between perceived target B and real target E. The resulting area intersection-union matrix is ​​as follows: or .

[0150] After calculating the angle intersection-comparison matrix and the area intersection-comparison matrix, the associated sensed targets and associated real targets can be determined based on the magnitude of each element in these matrices. The following example illustrates how to determine the associated sensed targets and associated real targets based on the magnitude of each element in both matrices, using the example of rows corresponding to sensed targets in both the angle and area intersection-comparison matrices.

[0151] When the rows of the angle intersection-over-union matrix correspond to the perceived target, the columns of the angle intersection-over-union matrix correspond to the real target. Similarly, when the rows of the area intersection-over-union matrix correspond to the perceived target, the columns of the area intersection-over-union matrix correspond to the real target. For example, the angle intersection-over-union matrix is... The area intersection-to-union matrix is .

[0152] In some implementations, for each perceived target in the set of perceived targets corresponding to the image data source type, a first target element greater than a first preset threshold is determined from the row corresponding to the perceived target in the angle intersection-over-union matrix. Then, the real targets corresponding to the columns containing each first target element in the angle intersection-over-union matrix are determined as candidate real targets. Second target elements corresponding to each candidate real target are determined from the row corresponding to the perceived target in the area intersection-over-union matrix. The maximum value among all second target elements is determined, and if the maximum value is greater than a second preset threshold, the candidate real target corresponding to the maximum value is determined as an associated real target, and the perceived target is determined as an associated perceived target. If the maximum value is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all candidate real targets; if the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as an associated real target, and the perceived target is determined as an associated perceived target.

[0153] The aforementioned intersection-to-union matrix Area intersection-to-union matrix Let's take an example to illustrate this.

[0154] For a perceived target A, a first target element greater than a first preset threshold is determined from the rows X(A,C), X(A,D), and X(A,E) corresponding to the perceived target A in the intersection-union matrix. If both X(A,C) and X(A,D) are greater than the first preset threshold, then the first target elements are X(A,C) and X(A,D).

[0155] The real targets corresponding to the columns containing the first target elements in the intersection-union matrix are determined as candidate real targets. Specifically, the real target C corresponding to the first column containing the first target element X(A, C) is determined as a candidate real target. The real target D corresponding to the second column containing the first target element X(A, D) is determined as a candidate real target.

[0156] From the rows Y(A,C), Y(A,D), and Y(A,E) corresponding to target A in the area intersection-union matrix, determine the second target elements Y(A,C) and Y(A,D) corresponding to each candidate true target C and D. Determine the maximum value from the second target elements Y(A,C) and Y(A,D). Assuming Y(A,C) is greater than Y(A,D), then the maximum value is Y(A,C).

[0157] If the maximum value Y(A, C) is greater than the second preset threshold, then the candidate real target C corresponding to the maximum value Y(A, C) is determined as an associated real target that is related to the perceived target A, and the perceived target A is determined as an associated perceived target.

[0158] If the maximum value Y(A, C) is less than or equal to the second preset threshold, then from all candidate real targets C and D, the first candidate real target with the smallest longitudinal error rate with the perceived target A is determined. The longitudinal error rate between the perceived target A and the candidate real targets C / D is calculated by calculating the difference between the Y coordinate of the center point of the perceived target A and the Y coordinate of the center point of the candidate real target C / D in the same coordinate system, and dividing this difference by the Y coordinate of the center point of the candidate real target C / D to obtain the longitudinal error rate between the perceived target A and the candidate real target C / D.

[0159] Assume the first candidate real target is D. If the longitudinal error rate between the first candidate real target D and the perceived target is less than a third preset threshold, then the first candidate real target D is determined as an associated real target related to the perceived target A, and the perceived target A is also determined as an associated perceived target. Conversely, if the longitudinal error rate between the first candidate real target D and the perceived target is greater than the third preset threshold, then the first candidate real target D and the perceived target A are not associated.

[0160] This method, which targets perceived targets within a set of perceived targets corresponding to an image data source type, first determines candidate real targets based on the intersection-union matrix (IU / U) of angles, and then identifies associated real targets from the candidate real targets based on the IU / U.S. area ...

[0161] However, if a smaller second preset threshold is set, when faced with nearby unrelated perceived targets and real targets, since the area intersection-union ratio of the nearby unrelated perceived targets and real targets is very small, it is possible that the nearby unrelated perceived targets and real targets are considered to be related.

[0162] Therefore, in this embodiment of the present disclosure, for the perceived targets in the set of perceived targets corresponding to the image data source type, candidate real targets are first determined based on the area intersection-union matrix. This filters out nearby unrelated perceived targets and real targets, avoiding the possibility that nearby unrelated perceived targets and real targets might be considered related due to their small area intersection-union ratio. Then, based on the area intersection-union matrix, associated real targets related to the perceived target are determined from the candidate real targets. This method makes it easier to set the second preset threshold and obtain more accurate association results.

[0163] The following explains how the intersection-union ratio (IU) of the angle between the perceived target and the real target is calculated.

[0164] Because the fusion perception algorithm transforms the point cloud data and the perceived image data into the same coordinate system, such as the vehicle coordinate system, before performing target recognition, each perceived target has a corresponding bounding box in the vehicle coordinate system that represents the position, size, and shape of the perceived target, i.e., the perceived target bounding box described in this disclosure. This bounding box can be determined based on the outer contour of the perceived target.

[0165] For a perceived target whose data source type is an image data source, the first included angle range of the corresponding perceived target bounding box in the target coordinate system can be calculated. Taking a rectangular bounding box as an example, the calculation method of the first included angle range is as follows: connect the four vertices of the perceived target bounding box to the origin of the target coordinate system, obtaining four lines. These four lines form angles with the X-axis of the target coordinate system, resulting in four included angles. The minimum value of the four included angles is determined as the lower limit of the first included angle range, and the maximum value of the four included angles is determined as the upper limit of the first included angle range, thus obtaining the first included angle range. Since these four lines form angles with the X-axis of the target coordinate system, the first included angle range determined based on these four included angles is the included angle range relative to the X-axis of the target coordinate system.

[0166] Similarly, the second included angle range corresponding to the real target bounding box in the target coordinate system can be calculated. Based on the first and second included angle ranges, the intersection-union ratio (IUGR) of the included angles between the perceived target and the real target can be calculated. The IUGR is the ratio of the intersection of the first and second included angle ranges to the union of the first and second included angle ranges.

[0167] The calculation method for the area intersection-union ratio (OU) of the perceived target and the real target is similar in principle to the calculation method for the angle intersection-union ratio (OU) of the perceived target and the real target. Here's a brief explanation: For a perceived target whose data source type is an image data source, calculate the first region corresponding to the perceived target's bounding box in the target coordinate system. Calculate the second region corresponding to the real target's bounding box in the target coordinate system. Determine the intersection region of the first region and the second region, and determine the union region of the first region and the second region. Calculate the ratio of the intersection region to the union region to obtain the area intersection-union ratio.

[0168] In the above-described implementation method, which associates real targets with each perceived target in the set of perceived targets corresponding to the image data source type according to the association processing strategy corresponding to the image data source type, and obtains the association processing sub-result corresponding to the set of perceived targets, a large amount of computation may be generated because the area intersection-union ratio between each perceived target and each real target is calculated for each perceived target in the set of perceived targets. To reduce the computational load, this disclosure also provides another implementation method that associates real targets with each perceived target in the set of perceived targets corresponding to the image data source type according to the association processing strategy corresponding to the image data source type, and obtains the association processing sub-result corresponding to the set of perceived targets corresponding to the image data source type. Specifically, it includes the following steps:

[0169] Step 1: For each of the perceived targets in the set of perceived targets, calculate the intersection-union ratio (IUU) between the perceived target and each of the real targets to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target.

[0170] For example, the intersection-union matrix of the included angle is .

[0171] Step 2: When the row of the angle intersection-union matrix corresponds to the sensing target, for each sensing target, determine a first target element greater than a first preset threshold from the row corresponding to the sensing target in the angle intersection-union matrix.

[0172] For example, for a perceived target A, a first target element greater than a first preset threshold is determined from the rows X(A,C), X(A,D), and X(A,E) corresponding to the perceived target A in the intersection-union matrix. If both X(A,C) and X(A,D) are greater than the first preset threshold, then the first target elements are X(A,C) and X(A,D).

[0173] Step 3: Determine the real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix as candidate real targets.

[0174] For example, the real target C corresponding to the first column where the first target element X (A, C) is located is determined as a candidate real target. The real target D corresponding to the second column where the first target element X (A, D) is located is determined as a candidate real target.

[0175] Step 4: Calculate the area intersection-union ratio of the perceived target and each of the candidate real targets, and determine the target area intersection-union ratio with the largest value.

[0176] For example, the area intersection-union ratio of perceived target A with candidate real targets C and D is calculated to obtain Y(A,C) and Y(A,D). Assuming that Y(A,C) is greater than Y(A,D), the target area intersection-union ratio is Y(A,C).

[0177] Step 5: If the intersection-union ratio of the target area is greater than the second preset threshold, the candidate real target corresponding to the intersection-union ratio of the target area is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0178] For example, if the intersection-union ratio of the target area Y(A,C) is greater than the second preset threshold, then the candidate real target C corresponding to the intersection-union ratio of the target area Y(A,C) is determined as an associated real target that is related to the perceived target A, and the perceived target A is determined as an associated perceived target.

[0179] Step 6: If the intersection-union ratio of the target area is less than or equal to the second preset threshold, determine the first candidate real target with the smallest longitudinal error rate with the perceived target from all the candidate real targets;

[0180] For example, if the intersection-union ratio of the target area Y(A,C) is less than or equal to the second preset threshold, then the first candidate real target with the smallest longitudinal error rate with the perceived target A is determined from all candidate real targets C and D.

[0181] Step 7: If the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as the associated real target that is mutually related to the perceived target, and the perceived target is determined as the associated perceived target.

[0182] For example, suppose the first candidate real target is D. If the longitudinal error rate between the first candidate real target D and the perceived target is less than a third preset threshold, then the first candidate real target D is determined to be an associated real target related to the perceived target A, and the perceived target A is also determined to be an associated perceived target. Conversely, if the longitudinal error rate between the first candidate real target D and the perceived target is greater than the third preset threshold, then the first candidate real target D and the perceived target A are not associated.

[0183] This approach avoids calculating the area intersection-union ratio (IU) between each perceived target and every real target in the set of perceived targets corresponding to the image data source type; instead, it only calculates the IU between the perceived target and candidate real targets. Since the number of candidate real targets may be less than the total number of real targets, this approach reduces computational cost.

[0184] Optionally, if the data source type of the sensing target set is a fusion data source type, the real target and each sensing target in the sensing target set are associated with each other according to the association processing strategy corresponding to the fusion data source type, to obtain the association processing sub-result corresponding to the sensing target set, including:

[0185] For each perceived target in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each real target is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns correspond to the other. For each perceived target in the set of perceived targets, the intersection-union ratio (IUU) of the area between the perceived target and each real target is calculated to obtain an area IUU matrix. The rows of the area IUU matrix correspond to one of the perceived target and the real target, and the columns correspond to the other. A target matrix is ​​obtained by weighted calculation based on the IUU matrix and the area IUU matrix. Based on the size of each element in the target matrix, associated perceived targets and associated real targets are determined. The associated processing sub-result includes the associated perceived targets and the associated real targets.

[0186] For example, suppose the set of perceived targets, whose data source type is image data source, contains two perceived targets A and B. All real targets are C, D, and E. Then, the intersection-union matrix can be calculated as follows: , or for The area intersection-to-union matrix is: or .

[0187] Furthermore, the target matrix is ​​obtained by weighting the intersection-union matrix of the included angle and the intersection-union matrix of the area. For example, [the following is a possible interpretation:] and Multiplying by the same digit yields the target matrix. .

[0188] Based on the target matrix The size of each element can determine the interconnected perceived targets and the interconnected real targets.

[0189] The following example illustrates how to determine the associated perceived targets and associated real targets based on the magnitude of each element in the target matrix, using the case where rows of both the angle intersection and comparison matrix and the area intersection and comparison matrix correspond to perceived targets. In this case, the rows of the target matrix correspond to perceived targets, and the columns of the target matrix correspond to real targets. Specifically, this includes the following steps:

[0190] Step 1: For each of the sensing targets in the set of sensing targets, determine the target element with the largest value from the row corresponding to the sensing target in the target matrix.

[0191] For example, for a perceived target A, from the target matrix In the rows Z(A,C), Z(A,D), and Z(A,E) corresponding to the perceived target A, the target element Z(A,C) with the largest value is determined.

[0192] Step 2: If the target element is greater than the fourth preset threshold, the real target corresponding to the column where the target element is located is determined as the associated real target that is related to the perceived target, and the perceived target is determined as the associated perceived target.

[0193] For example, if the target element Z (A, C) is greater than the fourth preset threshold, then the real target C corresponding to the column where the target element is located is determined as an associated real target that is related to the perceived target A, and the perceived target A is determined as an associated perceived target.

[0194] Step 3: If the target element is less than or equal to the fourth preset threshold, determine the first target element that is greater than the fifth preset threshold from the row corresponding to the perceived target in the angle intersection-union matrix.

[0195] For example, if the target element Z(A, C) is less than or equal to the fourth preset threshold, then the intersection-union matrix is ​​used. In the rows X(A,C), X(A,D), and X(A,E) corresponding to the perceived target A, determine the first target element that is greater than the fifth preset threshold. Assume the first target element is X(A,C) or X(A,D).

[0196] Step 4: Determine the real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix as the first candidate real targets.

[0197] For example, the real targets C and D corresponding to the columns containing the first target elements X(A, C) and X(A, D) are determined as the first candidate real targets.

[0198] Step 5: Calculate the confidence level between the perceived target and each of the first candidate real targets, and determine the second candidate real target corresponding to the confidence level that is greater than the sixth preset threshold.

[0199] Confidence level represents the probability that the perceived target and the real target are the same target (entity). The confidence level between the perceived target and the real target can be calculated by using the Mahalanobis distance between them. Furthermore, in calculating the Mahalanobis distance, the Gaussian distribution characteristics of the points in the point cloud acquired by the millimeter-wave radar sensor can be considered, and the positional errors of the points in the point cloud can be corrected according to the millimeter-wave radar position error model. Here, the Mahalanobis distance represents the covariance distance of the data. Mahalanobis distance is a distance metric that can be seen as a correction to Euclidean distance, correcting the problem of inconsistent and correlated scales across different dimensions in Euclidean distance. It is an effective method for calculating the similarity between two unknown sample sets.

[0200] In some implementations, a first sensing point in the sensing target frame of the sensing target that is closest to the vehicle carrying the millimeter-wave radar sensor and camera can be determined, and a second sensing point in the annotation frame of the first candidate real target that is closest to the vehicle carrying the millimeter-wave radar sensor and camera can be determined. The Mahalanobis distance between the first sensing point and the second sensing point is calculated to obtain the confidence level between the sensing target and the first candidate real target.

[0201] Based on the same principle, other sensing points in the sensing target bounding box of the sensing target and other sensing points in the labeled box of the first candidate real target can also be selected to calculate the Mahalanobis distance between them to obtain the similarity. This disclosure does not impose specific limitations on this.

[0202] Step 6: Determine the second candidate real target with the smallest longitudinal error rate with the perceived target as the associated real target that is mutually associated with the perceived target, and determine the perceived target as the associated perceived target.

[0203] For example, assuming that the first candidate real targets C and D are both second candidate real targets, then the second candidate real target with the smallest longitudinal error rate with the perceived target, assuming that the second candidate real target C is determined to be an associated real target that is related to the perceived target A, and the perceived target A is determined to be an associated perceived target.

[0204] Optionally, when the data source type of the sensing target set is a point cloud data source type, the real target and each sensing target in the sensing target set are associated according to the association processing strategy corresponding to the point cloud data source type, to obtain the association processing sub-result corresponding to the sensing target set, including:

[0205] For each perceived target in the set of perceived targets whose data source type is point cloud, the confidence score between the perceived target and each real target is calculated to obtain a confidence score matrix. The rows of the confidence score matrix correspond to one of the perceived target and the real target, and the columns of the confidence score matrix correspond to the other of the perceived target and the real target. The confidence score represents the probability that the perceived target and the real target are the same target.

[0206] Given that the rows of the confidence matrix correspond to the perceived target and the columns of the confidence matrix correspond to the real target, the first target element in the confidence matrix that is greater than the first threshold is determined.

[0207] The perceived target corresponding to the first target row where the first target element is located is identified as an associated perceived target; and the real target corresponding to the first target column where the first target element is located is identified as an associated real target. The associated perceived target and the associated real target are mutually associated.

[0208] For the second target row other than the first target row, determine the second target element with the highest confidence value in the second target row; determine the second target column where the second target element is located; if the second target element is the maximum value in the second target column, then determine the perceived target corresponding to the second target row as the associated perceived target; and determine the real target corresponding to the second target column as the associated real target.

[0209] For each real target, calculate the first angle between the real target and the X-axis in the target coordinate system based on the real target's position information. Determine the first angle interval based on the first angles corresponding to each real target. For candidate sensing targets other than the previously associated sensing targets, calculate the second angle between the candidate sensing target and the X-axis in the target coordinate system based on the candidate sensing target's position information. If the second angle is within the first angle interval, and there is a third target element in the third target row of the candidate sensing target that is greater than a second threshold, then the candidate sensing target is determined as an associated sensing target, where the second threshold is less than the first threshold. If the number of third target elements is greater than 1, then determine the candidate third target column corresponding to all third target elements. From the candidate real targets corresponding to all candidate third target columns, determine the target candidate real target that is closest to the candidate sensing target. This target candidate real target is determined as an associated real target.

[0210] In this case, a sensing target with a data source type of point cloud data source can correspond to multiple points in the point cloud data. Then, the point closest to the vehicle (i.e., the vehicle carrying the lidar sensor) can be determined from these multiple points to represent the sensing target. In other words, the point closest to the vehicle can be determined from these multiple points as the ground truth point of the sensing target.

[0211] For example, suppose there are two perceived targets A and B in the set of perceived targets corresponding to the point cloud data source type, and all real targets are C, D, and E. Then, for perceived target A, calculate the confidence score M(A, C) between perceived target A and real target C. Calculate the confidence score M(A, D) between perceived target A and real target D. Calculate the confidence score M(A, E) between perceived target A and real target E. Furthermore, for perceived target B, calculate the confidence score M(B, C) between perceived target B and real target C. Calculate the confidence score M(B, D) between perceived target B and real target D. Calculate the confidence score M(B, E) between perceived target B and real target E. The resulting confidence matrix is ​​as follows. or .

[0212] The confidence level represents the probability that the perceived target and the real target are the same target / entity. The confidence level between the perceived target and the real target can be calculated by calculating the Mahalanobis distance between them. Furthermore, in calculating the Mahalanobis distance, the Gaussian distribution characteristics of the points in the point cloud acquired by the millimeter-wave radar sensor can be considered, and the position error of the perceived target can be corrected according to the millimeter-wave radar position error model. It should be noted that the perceived target can be one or more points acquired by the millimeter-wave radar sensor; this embodiment uses a single point as an example for illustrative purposes.

[0213] For example, suppose the confidence matrix is... If the element M(A, D) in the first row and second column is greater than the first threshold, then the perceived target A corresponding to the first target element M(A, D) in the first row is determined as the associated perceived target. The real target D corresponding to the first target element M(A, D) in the second column is determined as the associated real target.

[0214] Furthermore, after determining the perceived target A as an associated perceived target and the real target D as an associated real target in the aforementioned embodiment, for the second target row (i.e., the second row where the associated perceived target A is located, excluding the first target row), the second target element with the largest confidence value in the second row is determined, assuming the second target element is M(B, E). The second target column (i.e., the third column of the confidence matrix) containing the second target element M(B, E) is determined. In the first case, if the second target element M(B, E) is the maximum value in the second target column (i.e., M(A, E) and M(B, E)), that is, if M(B, E) is greater than M(A, E), then the perceived target B corresponding to the second target row (i.e., the second row of the confidence matrix) is determined as an associated perceived target. And the real target E corresponding to the second target column (i.e., the third column of the confidence matrix) is determined as an associated real target.

[0215] In the second case, if the second target element M(B, E) is not the maximum value in the second target column, that is, M(A, E) is greater than M(B, E), then it is impossible to determine whether the perceived target corresponding to the row where the second target element M(B, E) is located is an associated perceived target, and it is impossible to determine whether the real target corresponding to the column where the second target element M(B, E) is located is an associated real target.

[0216] In the third case, if the second target element M(B, E) is not the maximum value in the second target column, that is, M(A, E) is greater than M(B, E), then the perceived target corresponding to the row where the second target element M(B, E) is located is determined to be an unassociated perceived target, and the real target corresponding to the column where the second target element M(B, E) is located is determined to be an unassociated real target.

[0217] The first angle between the real target and the X-axis in the target coordinate system refers to the angle between the first line connecting the coordinates of the real target and the origin of the target coordinate system and the X-axis.

[0218] Accordingly, the second angle between the candidate sensing target and the X-axis in the target coordinate system refers to the angle between the second line connecting the coordinates of the candidate sensing target and the origin of the target coordinate system and the X-axis.

[0219] The target coordinate system can be the vehicle coordinate system corresponding to the vehicle equipped with millimeter-wave radar sensors and cameras.

[0220] The following uses the confidence matrix as an example. Taking A as an example of an already associated sensing target, for real target C, the first angle a1 between real target C and the X-axis in the target coordinate system is calculated based on the position information of real target C. For real target D, the first angle a2 between real target D and the X-axis in the target coordinate system is calculated based on the position information of real target D. For real target E, the first angle a3 between real target E and the X-axis in the target coordinate system is calculated based on the position information of real target E. Based on the first angles a1, a2, and a3 corresponding to each real target C, D, and E, the first angle interval is determined, assumed to be [a1, a3]. For candidate sensing target B other than the already associated sensing target A, the second angle b between candidate sensing target B and the X-axis in the target coordinate system is calculated based on the position information of candidate sensing target B. In the first case, if the second angle b is within the first angle interval [a1, a3], that is, a... 1、 If b and a3, and there is a third target element in the third target row (i.e. the second row in the confidence matrix) where candidate perception target B is located that is greater than the second preset threshold, then candidate perception target B is determined as an associated perception target.

[0221] The following explains how to determine the associated real targets that are associated with the associated perceived target B when the candidate perceived target B is identified as an associated perceived target.

[0222] In one implementation, if the third target element has X (B, E), then the real target E corresponding to the column where the third target element X (B, E) is located is directly determined as the associated real target associated with the associated perceived target B.

[0223] In another implementation, if the number of third target elements is greater than 1, such as M(B, C), M(B, D), M(B, E), then the candidate third target columns corresponding to all third target elements are determined, i.e., the first, second, and third columns of the confidence matrix. From the candidate real targets C, D, and E corresponding to all candidate third target columns, the candidate real target closest to the candidate perceived target B is determined, assuming it is C. The candidate real target C is then determined as the associated real target related to the already associated perceived target B.

[0224] Optionally, determining the evaluation result of the fusion perception algorithm to be evaluated based on the association result includes:

[0225] The accuracy of the fusion sensing algorithm to be evaluated is determined based on the number of associated sensing targets and the total number of sensing targets.

[0226] The recall rate of the fusion perception algorithm to be evaluated is determined based on the number of associated real targets and the total number of real targets.

[0227] The evaluation results include the precision rate and the recall rate.

[0228] For example, assuming the number of associated perceived targets is 1 and the total number of perceived targets is 2, the precision of the fusion perception algorithm to be evaluated can be determined to be 50%. Based on the number of associated real targets is 2 and the total number of real targets is 3, the recall of the fusion perception algorithm to be evaluated can be determined to be 66.67%.

[0229] Optionally, the evaluation result of the fusion sensing algorithm to be evaluated is determined based on the association result, including:

[0230] Based on the association results, association pairs are determined. Each association pair includes the associated perceived target and the associated real target that is associated with the associated perceived target. The error between the associated perceived target and the associated real target in the association pair is calculated. Based on the error corresponding to each association pair, the mean error corresponding to each target category is calculated. The target category of the association pair is determined according to the category of the associated real target in the association pair. The evaluation results include the mean error.

[0231] For example, suppose that perceived target A is associated with real target C, and perceived target B is associated with real target E. Then there are two association pairs: A and C, and B and E. Calculate the error M between the associated perceived target A and the associated real target C in the association pair A and C. For example, the root mean square error (RMSE) of the position coordinates x and / or y of A and C, or the root mean square error of the velocity of A and C.

[0232] Calculate the root mean square error N between the associated perceived target B and the associated real target E in the association pair B and E.

[0233] Based on the errors corresponding to each association pair, the mean error for each target category is calculated. Assuming A and C correspond to the category of "cars," and B and E correspond to the category of "cars," then the mean error for the "cars" category is (M+N) / 2.

[0234] Target categories can also include large vehicles, pedestrians, birds, trees, billboards, etc.

[0235] Optionally, determining the evaluation result of the fusion sensing algorithm to be evaluated based on the association result may further include:

[0236] For association pairs with point cloud data sources, the confidence level of each association pair is determined. All association pairs are then sorted and plotted according to their confidence levels to obtain the first PR curve map. The evaluation results of the fusion perception algorithm to be evaluated include the first PR curve map.

[0237] The area under the PR curve represents the average precision (AP), which is an indicator used to evaluate the quality of an algorithm.

[0238] Accordingly, for each association pair of the fused data source, the Y-value of each association pair can be determined, and all association pairs can be sorted and plotted according to the Y-value to obtain the second PR curve map. The evaluation result of the fusion perception algorithm to be evaluated includes the second PR curve map. It should be noted that the above embodiment is an example in which the rows of the matrix correspond to the perceived target and the columns of the matrix correspond to the real target. Similarly, when the columns of the matrix correspond to the perceived target and the rows of the matrix correspond to the real target, the implementation method is essentially the same as the above embodiment, and will not be repeated here.

[0239] Figure 2 This is a block diagram illustrating an apparatus 200 for evaluating a fusion sensing algorithm according to an exemplary embodiment. (Refer to...) Figure 2 The device 200 for evaluating fusion perception algorithms includes:

[0240] The acquisition module 210 is configured to acquire point cloud data obtained by the millimeter-wave radar sensor perceiving the target scene, and the perceived image obtained by the camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view.

[0241] The first recognition module 220 is configured to perform target recognition on the point cloud data and the perception image according to the fusion perception algorithm to be evaluated, and obtain the recognized perception target;

[0242] The association module 230 is configured to associate the real target in the target scene with the perceived target to obtain an association result;

[0243] The execution module 240 is configured to determine the evaluation result of the fusion perception algorithm to be evaluated based on the association result.

[0244] Optionally, the number of perceived targets is multiple, the number of real targets is multiple, and the association module 230 includes:

[0245] The classification submodule is configured to classify all the sensing targets based on the data source type corresponding to each sensing target, and obtain a set of sensing targets corresponding to different data source types;

[0246] The association submodule is configured to, for each set of perception targets, perform association processing on the real target and each of the perception targets in the set of perception targets according to the association processing strategy corresponding to the data source type of the set of perception targets, to obtain the association processing sub-result corresponding to the set of perception targets; wherein, the association result includes the association processing sub-result corresponding to each set of perception targets.

[0247] Optionally, if the data source type of the perceived target set is an image data source type, the association submodule includes:

[0248] The first calculation submodule is configured to calculate the angle intersection ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an angle intersection ratio matrix, wherein the rows of the angle intersection ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection ratio matrix correspond to the other of the perceived target and the real target.

[0249] The second calculation submodule is configured to calculate the area intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an area intersection-union ratio matrix, wherein the rows of the area intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the area intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0250] The first determining submodule is configured to determine the associated perceived target and the associated real target based on the size of each element in the angle intersection-union matrix and the size of each element in the area intersection-union matrix. The association processing sub-result includes the associated perceived target and the associated real target.

[0251] Optionally, when a row of the included angle intersection-union matrix corresponds to the sensing target, and a row of the area intersection-union matrix corresponds to the sensing target, the first determining submodule is configured as follows:

[0252] For each perceived target, a first target element greater than a first preset threshold is determined from the row corresponding to the perceived target in the angle intersection-union matrix; and the real targets corresponding to the columns where each of the first target elements in the angle intersection-union matrix is ​​located are determined as candidate real targets; a second target element corresponding to each candidate real target is determined from the row corresponding to the perceived target in the area intersection-union matrix; the maximum value among all the second target elements is determined, and if the maximum value is greater than a second preset threshold, the candidate real target corresponding to the maximum value is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0253] Optionally, the first determining submodule is further configured to:

[0254] If the maximum value is less than or equal to the second preset threshold, a first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets; if the longitudinal error rate between the first candidate real target and the perceived target is less than the third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0255] Optionally, if the data source type of the perceived target set is an image data source type, the association submodule includes:

[0256] The third calculation submodule is configured to calculate the angle intersection ratio between the perceived target and each real target for each perceived target in the set of perceived targets, and obtain an angle intersection ratio matrix, wherein the rows of the angle intersection ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection ratio matrix correspond to the other of the perceived target and the real target.

[0257] The second determining submodule is configured to, when the rows of the angle intersection-union matrix correspond to the perceived targets, determine, for each perceived target, a first target element greater than a first preset threshold from the rows corresponding to the perceived targets in the angle intersection-union matrix; and determine the real targets corresponding to the columns where each of the first target elements in the angle intersection-union matrix is ​​located as candidate real targets.

[0258] The fourth calculation submodule is configured to calculate the area intersection-union ratio of the perceived target and each of the candidate real targets, and determine the target area intersection-union ratio with the largest value;

[0259] The third determining submodule is configured to, when the intersection-union ratio of the target area is greater than a second preset threshold, determine the candidate real target corresponding to the intersection-union ratio of the target area as the associated real target that is mutually associated with the perceived target, and determine the perceived target as the associated perceived target.

[0260] Optionally, the third determining submodule is further configured to:

[0261] If the intersection-union ratio of the target area is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets; if the longitudinal error rate between the first candidate real target and the perceived target is less than the third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0262] Optionally, the intersection-union ratio (IU) of the angle between the perceived target and the real target is calculated as follows:

[0263] Calculate the first included angle range of the perception target bounding box corresponding to the perception target in the target coordinate system. The perception target bounding box is determined based on the position of the perception target in the perception image.

[0264] Calculate the second included angle range of the real target bounding box corresponding to the real target in the target coordinate system, where both the first included angle range and the second included angle range are included angle ranges relative to the X-axis;

[0265] The intersection-union ratio of the angles between the perceived target and the real target is calculated based on the first angle range and the second angle range.

[0266] Optionally, if the data source type of the perceived target set is a fused data source type, the association submodule includes:

[0267] The fifth calculation submodule is configured to calculate the angle intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an angle intersection-union ratio matrix, wherein the rows of the angle intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the angle intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0268] The sixth calculation submodule is configured to calculate the area intersection-union ratio between the perceived target and each real target for each of the perceived targets in the set of perceived targets, and obtain an area intersection-union ratio matrix, wherein the rows of the area intersection-union ratio matrix correspond to one of the perceived target and the real target, and the columns of the area intersection-union ratio matrix correspond to the other of the perceived target and the real target.

[0269] The weighting submodule is configured to calculate the target matrix by weighting the included angle intersection-to-union matrix and the area intersection-to-union matrix.

[0270] The fourth determination submodule is configured to determine, based on the size of each element in the target matrix, the associated perceived targets and the associated real targets, and the associated processing sub-results include the associated perceived targets and the associated real targets.

[0271] Optionally, when a row of the included angle intersection-union matrix corresponds to the sensing target, and a row of the area intersection-union matrix corresponds to the sensing target, the fourth determining submodule is configured as follows:

[0272] For each of the perceived targets in the set of perceived targets, the target element with the largest value is determined from the row corresponding to the perceived target in the target matrix; if the target element is greater than a fourth preset threshold, the real target corresponding to the column where the target element is located is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target; if the target element is less than or equal to the fourth preset threshold, a first target element with a value greater than a fifth preset threshold is determined from the row corresponding to the perceived target in the intersection-union matrix; the real targets corresponding to the columns where each first target element is located in the intersection-union matrix are determined as first candidate real targets; the confidence level between the perceived target and each first candidate real target is calculated, and a second candidate real target corresponding to the confidence level greater than a sixth preset threshold is determined; the second candidate real target with the smallest longitudinal error rate with the perceived target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

[0273] Optionally, the apparatus 200 for evaluating the fusion perception algorithm further includes:

[0274] The second identification module is configured to perform target identification on the point cloud data according to the fusion perception algorithm to be evaluated, and obtain a first set of perception targets, wherein each first perception target in the first set of perception targets corresponds to a point cloud data source type;

[0275] The third recognition module is configured to perform target recognition on the perceived image according to the fusion perception algorithm to be evaluated, and obtain a second set of perceived targets, wherein each second perceived target in the second set of perceived targets corresponds to an image data source type;

[0276] The first determining module is configured to, for each of the perceived targets, determine a first perceived target with the highest similarity to the perceived target from the first set of perceived targets, and determine a second perceived target with the highest similarity to the perceived target from the second set of perceived targets;

[0277] The second determining module is configured to determine the data source type of the sensing target based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target.

[0278] Optionally, the second determining module includes:

[0279] The fifth determining submodule is configured to determine the data source type of the perceived target as a fusion data source type when the first similarity is greater than the similarity threshold and the second similarity is greater than the similarity threshold.

[0280] The sixth determining submodule is configured to determine the data source type of the perceived target as the image data source type when the first similarity is less than or equal to the similarity threshold and the second similarity is greater than the similarity threshold.

[0281] Optionally, the execution module 240 is configured to:

[0282] The precision of the fusion perception algorithm to be evaluated is determined based on the number of associated perceived targets and the total number of perceived targets; the recall of the fusion perception algorithm to be evaluated is determined based on the number of associated real targets and the total number of real targets; wherein the evaluation result includes the precision and the recall.

[0283] Using the aforementioned device, the merits of a fusion perception algorithm to be evaluated can be determined. The merits of a fusion perception algorithm to be evaluated can be characterized by metrics such as precision / accuracy and recall.

[0284] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0285] Those skilled in the art should understand that the device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and other division methods may exist in actual implementation. For instance, multiple modules may be combined or integrated into one module. Furthermore, the modules described as separate components may or may not be physically separated. Each module can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. When implemented in hardware, it can be implemented wholly or partially in the form of an integrated circuit or chip.

[0286] This disclosure also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the steps of the method for evaluating a fusion sensing algorithm provided in this disclosure.

[0287] Figure 3This is a block diagram illustrating a vehicle 600 according to an exemplary embodiment. For example, vehicle 600 can be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 600 can be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0288] Reference Figure 3 The vehicle 600 may include various subsystems, such as an infotainment system 610, a perception system 620, a decision control system 630, a drive system 640, and a computing platform 650. The vehicle 600 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 600 can be interconnected via wired or wireless means.

[0289] In some embodiments, the infotainment system 610 may include a communication system, an entertainment system, and a navigation system, etc.

[0290] The perception system 620 may include several sensors for sensing information about the environment surrounding the vehicle 600. For example, the perception system 620 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0291] The decision control system 630 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0292] The drive system 640 may include components that provide powered motion to the vehicle 600. In one embodiment, the drive system 640 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0293] Some or all of the functions of vehicle 600 are controlled by computing platform 650. Computing platform 650 may include at least one processor 651 and memory 652, processor 651 can execute instructions 653 stored in memory 652.

[0294] Processor 651 can be any conventional processor, such as a commercially available CPU. The processor may also include, for example, a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System on Chip (SOC), an Application Specific Integrated Circuit (ASIC), or a combination thereof.

[0295] The memory 652 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0296] In addition to instruction 653, memory 652 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 652 can be used by computing platform 650.

[0297] In this embodiment of the disclosure, processor 651 may execute instruction 653 to complete all or part of the steps of the above-described method for evaluating fusion perception algorithms.

[0298] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the method described above for evaluating the fusion perception algorithm when executed by the programmable device.

[0299] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0300] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for evaluating fusion perception algorithms, characterized in that, The method includes: The system acquires point cloud data obtained by a millimeter-wave radar sensor perceiving a target scene, and a perceived image obtained by a camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view. The point cloud data and the perceived image are used to perform target recognition based on the fusion perception algorithm to be evaluated, and the recognized perceived target is obtained. The real targets in the target scene are associated with the perceived targets to obtain an association result; the association process includes: classifying all the perceived targets based on the data source type corresponding to each perceived target to obtain a set of perceived targets corresponding to different data source types; For the perceived targets in the set of perceived targets corresponding to the image source data type, candidate real targets are first determined based on the angle intersection-union matrix, and then associated real targets that are mutually related to the perceived target are determined from the candidate real targets based on the area intersection-union matrix, and the perceived target is determined as an associated perceived target; wherein, the angle intersection-union matrix includes the angle intersection-union ratio between the perceived target and each of the real targets, and the area intersection-union matrix includes the area intersection-union ratio between the perceived target and each of the real targets; For the perceived targets in the set of perceived targets corresponding to the fusion source data type, a target matrix is ​​calculated by weighting the included angle intersection-union ratio matrix and the area intersection-union ratio matrix; based on the size of each element in the target matrix, the associated perceived targets and the associated real targets are determined. The evaluation result of the fusion perception algorithm to be evaluated is determined based on the association result, wherein the association result includes the association processing sub-results corresponding to each set of perception targets.

2. The method according to claim 1, characterized in that, When the data source type of the perceived target set is an image data source type, the method for obtaining the intersection-over-union matrix includes: For each of the perceived targets in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each of the real targets is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target. The method for obtaining the area intersection-union ratio matrix includes: for each of the sensing targets in the set of sensing targets, calculating the area intersection-union ratio between the sensing target and each of the real targets to obtain the area intersection-union ratio matrix, wherein the rows of the area intersection-union ratio matrix correspond to one of the sensing target and the real target, and the columns of the area intersection-union ratio matrix correspond to the other of the sensing target and the real target.

3. The method according to claim 2, characterized in that, The process of first determining candidate real targets based on the angle intersection-union matrix, then determining associated real targets from the candidate real targets based on the area intersection-union matrix, and finally identifying the perceived target as an associated perceived target includes: For each of the perceived targets, a first target element greater than a first preset threshold is determined from the row corresponding to the perceived target in the intersection-union matrix; and, The real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix are determined as candidate real targets; Determine the second target element corresponding to each candidate real target from the rows corresponding to the perceived target in the area intersection-union matrix; Determine the maximum value among all the second target elements, and if the maximum value is greater than a second preset threshold, determine the candidate real target corresponding to the maximum value as the associated real target that is mutually associated with the perceived target, and determine the perceived target as the associated perceived target.

4. The method according to claim 3, characterized in that, The method further includes: If the maximum value is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets. If the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

5. The method according to claim 1, characterized in that, When the data source type of the perceived target set is an image data source type, the method for obtaining the intersection-over-union matrix includes: For each of the perceived targets in the set of perceived targets, the intersection-union ratio (IUU) of the angle between the perceived target and each of the real targets is calculated to obtain an IUU matrix. The rows of the IUU matrix correspond to one of the perceived target and the real target, and the columns of the IUU matrix correspond to the other of the perceived target and the real target. When a row in the intersection-union-matrix (IUCM) corresponds to a sensing target, for each sensing target, a first target element greater than a first preset threshold is determined from the row corresponding to the sensing target in the IUCM; and, The real targets corresponding to the columns containing the first target elements in the angle intersection-union matrix are determined as candidate real targets; Calculate the area intersection-union ratio (IUU) of the perceived target and each of the candidate real targets, and determine the target with the largest IUU value; If the intersection-union ratio of the target area is greater than the second preset threshold, the candidate real target corresponding to the intersection-union ratio of the target area is determined as an associated real target that is mutually associated with the perceived target, and the perceived target is determined as an associated perceived target.

6. The method according to claim 5, characterized in that, The method further includes: If the intersection-union ratio of the target area is less than or equal to the second preset threshold, the first candidate real target with the smallest longitudinal error rate with the perceived target is determined from all the candidate real targets. If the longitudinal error rate between the first candidate real target and the perceived target is less than a third preset threshold, the first candidate real target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

7. The method according to claim 2, characterized in that, The intersection-union ratio of the angle between the perceived target and the real target is calculated in the following way: Calculate the first included angle range of the perception target bounding box corresponding to the perception target in the target coordinate system. The perception target bounding box is determined based on the position of the perception target in the perception image. Calculate the second included angle range of the real target bounding box corresponding to the real target in the target coordinate system, where both the first included angle range and the second included angle range are included angle ranges relative to the X-axis; The intersection-union ratio of the angles between the perceived target and the real target is calculated based on the first angle range and the second angle range.

8. The method according to claim 1, characterized in that, When the rows of the angle intersection-union matrix correspond to the perceived target, and the rows of the area intersection-union matrix correspond to the perceived target, determining the associated perceived target and the associated real target based on the size of each element in the target matrix includes: For each of the sensing targets in the set of sensing targets, determine the target element with the largest value from the row corresponding to the sensing target in the target matrix; When the target element is greater than the fourth preset threshold, the real target corresponding to the column where the target element is located is determined as the associated real target that is related to the perceived target, and the perceived target is determined as the associated perceived target; If the target element is less than or equal to the fourth preset threshold, a first target element greater than the fifth preset threshold is determined from the row corresponding to the perceived target in the angle intersection-union matrix. The real targets corresponding to the columns containing the first target elements in the angle intersection-comparison matrix are determined as the first candidate real targets; Calculate the confidence level between the perceived target and each of the first candidate real targets, and determine the second candidate real target corresponding to the confidence level that is greater than a sixth preset threshold; The second candidate real target with the smallest longitudinal error rate with the perceived target is determined as the associated real target that is mutually associated with the perceived target, and the perceived target is determined as the associated perceived target.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The point cloud data is used to identify targets according to the fusion perception algorithm to be evaluated, and a first set of perception targets is obtained. Each first perception target in the first set of perception targets corresponds to a point cloud data source type. The target recognition is performed on the perceived image according to the fusion perception algorithm to be evaluated, and a second set of perceived targets is obtained. The image data source type of each second perceived target in the second set of perceived targets is a different type of image data source. For each of the aforementioned sensing targets, a first sensing target with the highest similarity to the sensing target is determined from the first set of sensing targets, and a second sensing target with the highest similarity to the sensing target is determined from the second set of sensing targets; The data source type of the sensing target is determined based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target.

10. The method according to claim 9, characterized in that, The step of determining the data source type of the sensing target based on the magnitude of the first similarity between the sensing target and the first sensing target, and the magnitude of the second similarity between the sensing target and the second sensing target, includes: If the first similarity is greater than the similarity threshold and the second similarity is greater than the similarity threshold, the data source type of the perceived target is determined to be a fusion data source type. If the first similarity is less than or equal to the similarity threshold and the second similarity is greater than the similarity threshold, the data source type of the perceived target is determined to be the image data source type.

11. The method according to any one of claims 2-8, characterized in that, Determining the evaluation result of the fusion perception algorithm to be evaluated based on the association result includes: The accuracy of the fusion sensing algorithm to be evaluated is determined based on the number of associated sensing targets and the total number of sensing targets. The recall rate of the fusion perception algorithm to be evaluated is determined based on the number of associated real targets and the total number of real targets. The evaluation results include the precision rate and the recall rate.

12. An apparatus for evaluating fusion perception algorithms, characterized in that, The device includes: The acquisition module is configured to acquire point cloud data obtained by the millimeter-wave radar sensor perceiving the target scene, and the perceived image obtained by the camera perceiving the target scene, wherein the millimeter-wave radar sensor and the camera have the same perception field of view. The first recognition module is configured to perform target recognition on the point cloud data and the perception image according to the fusion perception algorithm to be evaluated, and obtain the recognized perception target; The association module is configured to associate real targets in the target scene with the perceived targets to obtain an association result; the association process includes: classifying all the perceived targets based on the data source type corresponding to each perceived target to obtain a set of perceived targets corresponding to different data source types; For the perceived targets in the set of perceived targets corresponding to the image source data type, candidate real targets are first determined based on the angle intersection-union matrix, and then associated real targets that are mutually related to the perceived target are determined from the candidate real targets based on the area intersection-union matrix, and the perceived target is determined as an associated perceived target; wherein, the angle intersection-union matrix includes the angle intersection-union ratio between the perceived target and each of the real targets, and the area intersection-union matrix includes the area intersection-union ratio between the perceived target and each of the real targets; For the perceived targets in the set of perceived targets corresponding to the fusion source data type, a target matrix is ​​calculated by weighting the included angle intersection-union ratio matrix and the area intersection-union ratio matrix; based on the size of each element in the target matrix, the associated perceived targets and the associated real targets are determined. The execution module is configured to determine the evaluation result of the fusion perception algorithm to be evaluated based on the association result, wherein the association result includes the association processing sub-results corresponding to each set of perception targets.

13. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1 to 11.

14. A vehicle, characterized in that, include: A memory on which computer programs are stored; A processor for executing the computer program in the memory to implement the steps of the method according to any one of claims 1 to 11.

Images